Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Next generation Unix pipe by Alex Larsson

blogs.gnome.org

Post reply on HN
Print view View original

31–40 of 86 posts

Re: Next generation Unix pipe by Alex Larsson

#31

Post by luriel »

comex wrote: post #15
rhizome wrote: post #3

If I can do a slight PG impression, "what problem does this solve?"

Among others, this problem: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. Or a little recurring problem I have: How do I grep the output of grep -C (matches showing multiple lines delimited with a "--" line)? I wrote a custom tool to do it, which does the job, but really it w…

> find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell.

This problem is simply a flaw in sh (and its descendants), other shells handle it much better, see for example Tom Duff's rc shell: http://rc.cat-v.org

Also note that Plan 9, the successor to Unix (and which uses the rc shell as its main shell) doesn't even have a find command, find's design is not really very unix-y.

As for your second questions, the answer might be structural regular expressions: http://doc.cat-v.org/bell_labs/structural_regexps/

Re: Next generation Unix pipe by Alex Larsson

#32

Post by alexlarsson »

dfc wrote: post #17
comex wrote: post #14

Earlier quoted context omitted.

Two problems: the header is sorted along with the fields, and you have to look up the field number. Insurmountable? No; but somewhat complicated.

`grep -v` or `tail` seems like a lot easier workaround than desiging a brand new shell piping system. Maybe there are other use cases but sorting numeric fields is definitely not worth the effort. A lot of black magic can be easily conjured up with `col`, `tr`, `cut`, `column`, `head`, `tail`, `grep`, `pr`, and `sort`; and that's without ever even touching `sed` and `awk`.

Obviously a lot can be done, but its hardly easy, you yourself call it black magic. But its very easy to do with typed data, and only the beginning of what you can do.

Re: Next generation Unix pipe by Alex Larsson

#34

Post by alexlarsson »

sprobertson wrote: post #23

I've been playing around with a similar idea - using plain JSON as the message format, you can make a set of pipeable command line utilities for manipulating data from many web APIs.

A textual format needs parsing though at every stage in the pipeline. Thats why I think using an (optional, when supported) binary format is important.

Re: Next generation Unix pipe by Alex Larsson

#35

Post by alexlarsson »

comex wrote: post #15
rhizome wrote: post #3

If I can do a slight PG impression, "what problem does this solve?"

Among others, this problem: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. Or a little recurring problem I have: How do I grep the output of grep -C (matches showing multiple lines delimited with a "--" line)? I wrote a custom tool to do it, which does the job, but really it w…

The GVariant/dbus typesystem has both "string" which is a UTF8 text string and "bytestring" which is an array of bytes. The later is what you would use for filenames and would avoid problems with weird characters in filenames, etc.

Re: Next generation Unix pipe by Alex Larsson

#36

Post by dfc »

alexlarsson wrote: post #32
dfc wrote: post #17

Earlier quoted context omitted.

`grep -v` or `tail` seems like a lot easier workaround than desiging a brand new shell piping system. Maybe there are other use cases but sorting numeric fields is definitely not worth the effort. A lot of black magic can be easily conjured up with `col`, `tr`, `cut`, `column`, `head`, `tail`, `grep`, `pr`, and `sort`; and that's without ever even touching `sed` and `awk`.

Obviously a lot can be done, but its hardly easy, you yourself call it black magic. But its very easy to do with typed data, and only the beginning of what you can do.

When I said black magic[1] the last thing I was trying to convey was that using coreutils/bsdmainutils was complicated. Typed data is easy to work with, but creating a sophisticated unix pipes 2.0 is not. No matter how complicated you think coreutils/bsdmainutils mastery is, you have to admit its a lot easier than building unix pipes 2.0.

If you throw in numutils and moreutils you can go nuts with columns of data. What tasks would you like to accomplish on the command line with columns of typed data and pipes 2.0?

[1] On a side note I was surprised that we had different conceptions of what black magic. I was going for evil, nefarious, and/or unorthodox. Have I been using the term wrong? That's an honest question, it would not surprise me if I have been oblivious.

Re: Next generation Unix pipe by Alex Larsson

#37

Post by dfc »

rogerbinns wrote: post #22

What would be ideal to solve first is some sort of initial format negotiation on pipes. Otherwise you will end up with the wrong thing happening (eg having to reimplement every tool, spewing "rich" format to tools that don't know it, or regular text to tools that could do better). We've already seen something like this - for example ls does column output if going directly to a screen, otherwise one per line, and many…

man grep:

   -Z, --null
      Output a zero byte (the ASCII NUL character) instead  of  the  character  that  normally
      follows  a  file  name.   For example, grep -lZ outputs a zero byte after each file name
      instead of the usual newline.  This option makes the output  unambiguous,  even  in  the
      presence  of file names containing unusual characters like newlines.  This option can be
      used with commands like find -print0,  perl  -0,  sort  -z,  and  xargs  -0  to  process
      arbitrary file names, even those that contain newline characters.

   -z, --null-data
      Treat the input as a set of lines, each  terminated  by  a  zero  byte  (the  ASCII  NUL
      character)  instead of a newline.  Like the -Z or --null option, this option can be used
      with commands like sort -z to process arbitrary file names.
man xargs:

   --null
   -0     Input  items are terminated by a null character instead of by whitespace, and the quotes
      and backslash are not special (every character is taken literally).  Disables the end of
      file  string,  which  is treated like any other argument.  Useful when input items might
      contain white space, quote marks, or backslashes.  The GNU find -print0 option  produces
      input suitable for this mode.
man find:

   -print0
      True;  print  the  full  file  name on the standard output, followed by a null character
      (instead of the newline character that -print uses).  This allows file names  that  con‐
      tain newlines or other types of white space to be correctly interpreted by programs that
      process the find output.  This option corresponds to the -0 option of xargs.

Re: Next generation Unix pipe by Alex Larsson

#38

Post by alexlarsson »

snprbob86 wrote: post #5

Neat! I've commented about this very problem before on several of the many threads regarding "object pipes", ie. REPLs. http://news.ycombinator.com/item?id=1033623 http://news.ycombinator.com/item?id=1566325 http://news.ycombinator.com/item?id=2527217 Since that last comment, I've been working a bunch with Clojure, which has a far more expressive variant of JSON, as well as some heavy duty work with Google's Protocol…

Regarding order. The dtools approach uses a stream (i.e. potentially infinite) of variants. Each variant is a self contained typed data chunk which is by itself not "streamable" (i.e. you have to read all of it). The data chunk is strongly typed and the type is self-described.

The supported primitive types are: bool, byte, int16, uint16, int32, uint32, int64, uint64, double, utf8 string (+ some dbus specific things).

These can be recursively combined with: arrays (of same type), tuples, dicts (primitive type -> any type map), maybe type, and variant type

In my dps example I generate a stream of dictionaries mapping from string to variant (i.e. any type). The type of each item in the map differs. For instance cmdvec is an array of strings, whereas euid is an uint32.

Re: Next generation Unix pipe by Alex Larsson

#39

Post by comex »

luriel wrote: post #31
comex wrote: post #15

Earlier quoted context omitted.

Among others, this problem: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. Or a little recurring problem I have: How do I grep the output of grep -C (matches showing multiple lines delimited with a "--" line)? I wrote a custom tool to do it, which does the job, but really it w…

> find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. This problem is simply a flaw in sh (and its descendants), other shells handle it much better, see for example Tom Duff's rc shell: http://rc.cat-v.org Also note that Plan 9, the successor to Unix (and which uses the rc shell as its main shell) doesn't even have a find command, find's design…

> This problem is simply a flaw in sh (and its descendants)

Indeed, although it's not just sh; if you want to, say, make a table of filenames and some attributes of each file, you're in trouble if the filenames contain spaces (awk, cut, sort don't work as easily) and screwed if they contain newlines.

What does Plan 9 use instead of find?

> As for your second questions, the answer might be structural regular expressions:

I've actually been meaning to write a clone of the command line portion of sam, tack on some slightly more powerful features, and try living with it... it would be able to solve much of that use case, but I think it would be cleaner if all the normal tools just knew that the output of grep -C is, in fact, a list of multiline strings.

Re: Next generation Unix pipe by Alex Larsson

#40

Post by alexlarsson »

dfc wrote: post #37
rogerbinns wrote: post #22

What would be ideal to solve first is some sort of initial format negotiation on pipes. Otherwise you will end up with the wrong thing happening (eg having to reimplement every tool, spewing "rich" format to tools that don't know it, or regular text to tools that could do better). We've already seen something like this - for example ls does column output if going directly to a screen, otherwise one per line, and many…

man grep: -Z, --null Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name. For example, grep -lZ outputs a zero byte after each file name instead of the usual newline. This option makes the output unambiguous, even in the presence of file names containing unusual characters like newlines. This option can be used with commands like find -print0, perl -0, sort -z, and…

Yes, in other words, the parent is right that zero termination is currently not automatic.
Post reply on HN