Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Next generation Unix pipe by Alex Larsson

blogs.gnome.org

Post reply on HN
Print view View original

41–50 of 86 posts

Re: Next generation Unix pipe by Alex Larsson

#41

Post by dfc »

alexlarsson wrote: post #40
dfc wrote: post #37

Earlier quoted context omitted.

man grep: -Z, --null Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name. For example, grep -lZ outputs a zero byte after each file name instead of the usual newline. This option makes the output unambiguous, even in the presence of file names containing unusual characters like newlines. This option can be used with commands like find -print0, perl -0, sort -z, and…

Yes, in other words, the parent is right that zero termination is currently not automatic.

He did not say automatic, he said "knew about" nulls. When you talk about automagically detecting nulls I have this image of an ascii-art Clippy with a cowsay bubble that says "I see you are using null terminated data, I have enabled --null for you."

Re: Next generation Unix pipe by Alex Larsson

#42

Post by alexlarsson »

dfc wrote: post #36
alexlarsson wrote: post #32

Earlier quoted context omitted.

Obviously a lot can be done, but its hardly easy, you yourself call it black magic. But its very easy to do with typed data, and only the beginning of what you can do.

When I said black magic[1] the last thing I was trying to convey was that using coreutils/bsdmainutils was complicated. Typed data is easy to work with, but creating a sophisticated unix pipes 2.0 is not. No matter how complicated you think coreutils/bsdmainutils mastery is, you have to admit its a lot easier than building unix pipes 2.0. If you throw in numutils and moreutils you can go nuts with columns of data. Wh…

I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once.

And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation like:

filter out all processes with %cpu > 20 with uid > 1000 and sort by second cmdline arg as:

dps | dfilter pcpu ">" 20 uid ">" 1000 | dsort "cmdvect[1]"

Obviously a made up example, but something like this is easy to read and write, whereas something working on tabular ascii data would be quite long and complicated.

As per black magic, I have about the same interpretation as you. I didn't really misunderstand it to be about how complicated it was. However, "black magic" certainly has a feel of "you should not do this", and arguing that you can then use that in order to do something which could instead be simple and obvious in a typed system seems kind of weird.

Re: Next generation Unix pipe by Alex Larsson

#43

Post by alexlarsson »

dfc wrote: post #41
alexlarsson wrote: post #40

Earlier quoted context omitted.

Yes, in other words, the parent is right that zero termination is currently not automatic.

He did not say automatic, he said "knew about" nulls. When you talk about automagically detecting nulls I have this image of an ascii-art Clippy with a cowsay bubble that says "I see you are using null terminated data, I have enabled --null for you."

"automatically knew", in a post which talks about format negotiation. It was fairly obvious to me he meant that it would use the format negotiation to automatically enable the --null switch.

Re: Next generation Unix pipe by Alex Larsson

#44

Post by dfc »

alexlarsson wrote: post #42
dfc wrote: post #36

Earlier quoted context omitted.

When I said black magic[1] the last thing I was trying to convey was that using coreutils/bsdmainutils was complicated. Typed data is easy to work with, but creating a sophisticated unix pipes 2.0 is not. No matter how complicated you think coreutils/bsdmainutils mastery is, you have to admit its a lot easier than building unix pipes 2.0. If you throw in numutils and moreutils you can go nuts with columns of data. Wh…

I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once. And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation lik…

As you admit your system is not simple to implement but you are correct the onus is on you to create pipes 2.0. But it also means that every other developer is going to have to implement pipes 2.0 compliant output for their program. Unless pipes 2.0 is going to auto-identify everything in addition to nulls?

Re: Next generation Unix pipe by Alex Larsson

#45

Post by ibotty »

alexlarsson wrote: post #40
dfc wrote: post #37

Earlier quoted context omitted.

man grep: -Z, --null Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name. For example, grep -lZ outputs a zero byte after each file name instead of the usual newline. This option makes the output unambiguous, even in the presence of file names containing unusual characters like newlines. This option can be used with commands like find -print0, perl -0, sort -z, and…

Yes, in other words, the parent is right that zero termination is currently not automatic.

null-terminated strings are hard to read in a shell window. and isatty(3) does not work for pagers.

content nagotiation only works with bi-directional data transfer (i.e. not with pipes).

Re: Next generation Unix pipe by Alex Larsson

#46

Post by alexlarsson »

ibotty wrote: post #45
alexlarsson wrote: post #40

Earlier quoted context omitted.

Yes, in other words, the parent is right that zero termination is currently not automatic.

null-terminated strings are hard to read in a shell window. and isatty(3) does not work for pagers. content nagotiation only works with bi-directional data transfer (i.e. not with pipes).

Thats only true if you only negotiate via data in the pipe. dtools (in the article) uses non-mandatory file locks to do the content negotiation on the pipe.

Re: Next generation Unix pipe by Alex Larsson

#47

Post by alexlarsson »

dfc wrote: post #44
alexlarsson wrote: post #42

Earlier quoted context omitted.

I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once. And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation lik…

As you admit your system is not simple to implement but you are correct the onus is on you to create pipes 2.0. But it also means that every other developer is going to have to implement pipes 2.0 compliant output for their program. Unless pipes 2.0 is going to auto-identify everything in addition to nulls?

Yes, the weak link here is obviously getting all kinds of output into the pipes 2.0 format. Triggering such output via format negotiation is possible, but you still would have to add support for actually outputting it.

Re: Next generation Unix pipe by Alex Larsson

#49

Post by mongol »

For certain types of unix pipeing, I have found it useful to pipe from tool to CSV, and then let sqlite process the data, using SQL statements. SQL solves many of the sorting, filtering, joining things that you can do with unix pipes too, but with a syntax that is broadly known. Especially the joining I have found hard to do well with shell/piping.

I think a sqlite-aware shell would be awesone, especially if common tools had a common output format (like csv with header) where that also included the schema / data format.

Post reply on HN