Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Next generation Unix pipe by Alex Larsson

blogs.gnome.org

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Next generation Unix pipe by Alex Larsson

#61

Post by yason »

I wonder where this comes from. There's no need for next generation.

People have, for thirty years or so, successfully printed their data into a suitable textual streams for processing with programs glued together with pipes, and optionally parsed the results back into some native format if so required.

Meanwhile, none of the "next generation" pipes have gained any momentum. Obviously they solve something which is either not a problem or they do solve some problems but create new ones in greater numbers than what they solved, tipping the balance into negative.

Any object or intermediary format you can think of can be represented in text, and you're back to square one. For example, even if there's XSLT and XQuery, you can just serialize trees of XML elements into a row-based format of expression and use grep on the resulting stream to effectively make hierarchical searches inside the tree.

Re: Next generation Unix pipe by Alex Larsson

#62

Post by chris_wot »

Groxx wrote: post #20
chris_wot wrote: post #7

What about providing a filter that converts to whatever format you can think of? e.g. outputs in JSON or XML

Because what are you converting from ? It can't be turtles all the way down, at some point there must be a defined system that everything speaks. Adding output formats after that is relatively simple.

The output is typed, in a standard format. He defines in his post (cf output of the dfs program).

Re: Next generation Unix pipe by Alex Larsson

#63

Post by chris_wot »

rbanffy wrote: post #13

I find the tendency to repeat Microsoft's mistakes deeply disturbing. Even if, in this case, the author acknowledges PoweeShell goes too far, his own idea goes too far. I'd be all in with flags that make ps or ls spit JSON or XML, but this typed nonsense? What when I want to output a color? Will I need a new type? Oh... and the sort thing... its not hard to sort numerically.

What do you mean by "what when I want to output a color"?

Re: Next generation Unix pipe by Alex Larsson

#64

Post by a3_nm »

dfc wrote: post #36
alexlarsson wrote: post #32

Earlier quoted context omitted.

Obviously a lot can be done, but its hardly easy, you yourself call it black magic. But its very easy to do with typed data, and only the beginning of what you can do.

When I said black magic[1] the last thing I was trying to convey was that using coreutils/bsdmainutils was complicated. Typed data is easy to work with, but creating a sophisticated unix pipes 2.0 is not. No matter how complicated you think coreutils/bsdmainutils mastery is, you have to admit its a lot easier than building unix pipes 2.0. If you throw in numutils and moreutils you can go nuts with columns of data. Wh…

Black magic is "A technique that works, though nobody really understands why." http://www.catb.org/jargon/html/B/black-magic.html

Re: Next generation Unix pipe by Alex Larsson

#65

Post by Someone »

veyron wrote: post #12

That is a terrible idea: sometimes the app can take advantage of a constraint to minimize work done. In your example, if we just wanted to filter for a particular user, dps would have to print out ALL of the information and then you could pick at it. This doesn't seem bad for ps (because there's a hard limit) but in many other examples the output could be much larger than what is needed. That's why having filtering a…

If "sometimes the app can take advantage of a constraint" is an argument here, you should be against all usage of pipes.

Re: Next generation Unix pipe by Alex Larsson

#66

Post by pjmlp »

manojlds wrote: post #58
pjmlp wrote: post #56

Actually I prefer Powershell's approach to transfer objects, as it is more flexible than standardize in a specific transfer format. But I do concede that it has the downside that if the object lacks the properties you want to access, then it might be painful in some cases.

But it is not like you cannot do normal string processing using cmdlets like "Select-String". And an object missing a property is almost same a column missing in the returned text output right?

Good point, I've forgotten about that.

Re: Next generation Unix pipe by Alex Larsson

#67

Post by pka »

I can't believe this. Just 2 or so weeks ago I set about writing exactly something like this in Haskell [1]. It's by no means complete or even working at this point, but basically what I had in mind was something like:

    yls | yfilter 'mdate = yesterday && permissions.oread = true' | yformat -ls
Every tool emits or consumes "typed" JSON (i.e. JSON data with an additional JSON schema). Why typed? Because then the meaning of things like mdate = yesterday can be inferred from the type of mdate and mean different things depending on whether mdate is a string or a date. In the case of a date, the expression mdate = yesterday can automatically be rewritten to mdate >= 201208110000 && mdate etc. In the case of a string we do string comparison. In the case of a bool we emit an error if the compared-to value isn't either true or false, etc.

Basically, I wanted to build a couple of standard tools inspired by the FP world, like filter, sort, map, fold (reduce) and have an universal tool for outputting formatted data in whatever form is desired - be it JSON, csv files, text files or custom formats. Every tool would support an -f parameter, which means that its output is automatically piped through the format tool, so that something like

    yls -fls
is functionally equivalent to

    yls | yformat -ls
which would output the JSON data from yls in the traditional ls way on a unix system.

    yls | yformat -csv
would output csv data. Some more examples:

    yls | yfold '+ size' 0
prints out the combined size of all files in the current directory.

    yls | ymap 'name = name + .jpg' | ymv
would append .jpg to all files in the current directory.

    ycontacts | yfilter -fcsv 'name = *John*'
would print out all Google contacts containing John in their name as a csv file.

    yps | yfilter 'name = java*' | yeval 'kill name'
would kill all processes whose names start with 'java'.

The cool thing about this is that this approach conserves one of the main selling points of FP: composability. I.e. you can throw something like yfold '+ size' 0 in a shell script and then write:

    yls | size.sh
This way people would be able to build an ever growing toolbelt of abstracted functionality specifically tailored to their way of doing things, without losing composability.

[1] https://github.com/pkamenarsky/ytools

Re: Next generation Unix pipe by Alex Larsson

#68

Post by veyron »

Someone wrote: post #65
veyron wrote: post #12

That is a terrible idea: sometimes the app can take advantage of a constraint to minimize work done. In your example, if we just wanted to filter for a particular user, dps would have to print out ALL of the information and then you could pick at it. This doesn't seem bad for ps (because there's a hard limit) but in many other examples the output could be much larger than what is needed. That's why having filtering a…

If "sometimes the app can take advantage of a constraint" is an argument here, you should be against all usage of pipes.

That's not true. So in the case of `ps`, there is a known limit to the number of processes, and it is fairly small, so the performance hit is limited.

As another example in this context, if the original data source is gzip'd, it's faster to gunzip and then pipe rather than integrating the gzip logic into the app itself.

Re: Next generation Unix pipe by Alex Larsson

#69

Post by vidarh »

alexlarsson wrote: post #42
dfc wrote: post #36

Earlier quoted context omitted.

When I said black magic[1] the last thing I was trying to convey was that using coreutils/bsdmainutils was complicated. Typed data is easy to work with, but creating a sophisticated unix pipes 2.0 is not. No matter how complicated you think coreutils/bsdmainutils mastery is, you have to admit its a lot easier than building unix pipes 2.0. If you throw in numutils and moreutils you can go nuts with columns of data. Wh…

I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once. And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation lik…

Your example would be about the same length with awk and sort, with the only caveat that you need to figure out the field numbers, and the upside that I can trust the tools are available pretty much everywhere.
Post reply on HN