Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Next generation Unix pipe by Alex Larsson

blogs.gnome.org

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Next generation Unix pipe by Alex Larsson

#71

Post by ufo »

yason wrote: post #61

I wonder where this comes from. There's no need for next generation. People have, for thirty years or so, successfully printed their data into a suitable textual streams for processing with programs glued together with pipes, and optionally parsed the results back into some native format if so required. Meanwhile, none of the "next generation" pipes have gained any momentum. Obviously they solve something which is ei…

I have the opposite experience, actually. Its so damn annoying to have programs communicating structured data with each other over pipes that more complicated things inevitably diverge into a) monocultural programs made with a single programming language and that don't communicate with the outside world or b) Some form of exchange format (JSON, XMl, etc) that needs to be explicitly supported by every participant.

And unix utilities suck at handling structured data. If your file format is line based you might have a chance of it being easy to work with but don't even ask what happens if you insert a newline inside a textfield then.

Re: Next generation Unix pipe by Alex Larsson

#72

Post by 0xbadcafebee »

My preference for a "next generation pipe": Shared file descriptors. (Sort of)

It would work virtually the same as a standard pipe; the difference being you could control whether it was read, write, or both, and every application you 'piped' to would have access to the same file descriptors as the parent, unless a process in the path of the pipe closes one.

The end result will be the equivalent of passing unlimited individual arbitrary bitstreams combined with the ability to chain arbitrary programs. In fact, you could simplify things by simply passing the previous piped command's output as a new file descriptor to the next program, so you could easily reference the last piped program's output, or any of the ones before it.

For example:

cat arbitrary_data.docx | docx --count-words

STREAM[0] is the output of 'cat'. STREAM[1] is the counted words of STREAM[0] ($S[0] is an alias). STREAM[2] is the first 4 lines of the doc. STREAM[3] is the last 4 lines of the doc. STREAM[4] is the counted words from STREAM[2] (note the "). And STREAM[5] is the output of 'echo', though since it's the last command, it becomes STDOUT.

There may be a more slick way of doing this, but you can see the idea. Pass arbitrary streams as you pipe, and reference any of them at any point in the pipe to continue processing data arbitrarily in a one-liner.

...

Actually, it looks like this is already built into bash (sort of), as the Coprocesses functionality. I don't know if you can use it with pipes, but it's very interesting.

Re: Next generation Unix pipe by Alex Larsson

#73

Post by Daniel_Newby »

veyron wrote: post #12

That is a terrible idea: sometimes the app can take advantage of a constraint to minimize work done. In your example, if we just wanted to filter for a particular user, dps would have to print out ALL of the information and then you could pick at it. This doesn't seem bad for ps (because there's a hard limit) but in many other examples the output could be much larger than what is needed. That's why having filtering a…

So the NASDAQ dumper should accept a structured query as its input. This is an architecture issue, not a data format issue.

Re: Next generation Unix pipe by Alex Larsson

#74

Post by njharman »

comex wrote: post #15
rhizome wrote: post #3

If I can do a slight PG impression, "what problem does this solve?"

Among others, this problem: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. Or a little recurring problem I have: How do I grep the output of grep -C (matches showing multiple lines delimited with a "--" line)? I wrote a custom tool to do it, which does the job, but really it w…

filenames with newlines is an edgecase and a data problem. You can have fucked up characters in filenames doesn't mean you should.

You can spend 10 hours solving for edge cases or 1 hour redefining the problem. In this case by mandating we only work on files with ascii printable characters. It's often much, much easier to massage input than have follow on tools handle every freaking possible edge case ever.

Re: Next generation Unix pipe by Alex Larsson

#75

Post by Daniel_Newby »

huhtenberg wrote: post #18

At the risk of stating the obvious - this won't take off for a simple reason of being too complex by Unix standards .

Have you seen the command line options for ps? They're going to have to start using Unicode accent marks if they extend it much further.

Re: Next generation Unix pipe by Alex Larsson

#76

Post by rogerbinns »

dfc wrote: post #41
alexlarsson wrote: post #40

Earlier quoted context omitted.

Yes, in other words, the parent is right that zero termination is currently not automatic.

He did not say automatic, he said "knew about" nulls. When you talk about automagically detecting nulls I have this image of an ascii-art Clippy with a cowsay bubble that says "I see you are using null terminated data, I have enabled --null for you."

I did exactly say automatic. It is the previous word to "knew about" you quoted!

And yes, I would expect that find detects that when it is talking to xargs then null termination should be used without the user having to go and fish out what the options are for each tool. And if you used ps with another tool that prefers json then ps can automatically do that, again without having to find and maintain flags.

Re: Next generation Unix pipe by Alex Larsson

#77

Post by rogerbinns »

alexlarsson wrote: post #28
rogerbinns wrote: post #22

What would be ideal to solve first is some sort of initial format negotiation on pipes. Otherwise you will end up with the wrong thing happening (eg having to reimplement every tool, spewing "rich" format to tools that don't know it, or regular text to tools that could do better). We've already seen something like this - for example ls does column output if going directly to a screen, otherwise one per line, and many…

My code does format negotiation on the pipe to determine whether to send the data in textual form or binary form. It uses file locks (F_SETLK) on the pipe with a magic offset value offset to do the negotiation.

But you still have race conditions. The sender would have to ensure that the locks are setup before the receiver calls read() for the first time. Since pipes are often setup by the shell you have no control over the startup times. Sure you could have heuristics such as the receiver waiting a few seconds just in case locks show up, but that just makes things slow and unpredictable.

I stand by my assertion that this can only be solved well (ie 100% predictable behaviour no matter what order things start in or how long they take to intialise) by a new system call/ioctl.

Re: Next generation Unix pipe by Alex Larsson

#78

Post by alexlarsson »

rogerbinns wrote: post #77
alexlarsson wrote: post #28

Earlier quoted context omitted.

My code does format negotiation on the pipe to determine whether to send the data in textual form or binary form. It uses file locks (F_SETLK) on the pipe with a magic offset value offset to do the negotiation.

But you still have race conditions. The sender would have to ensure that the locks are setup before the receiver calls read() for the first time. Since pipes are often setup by the shell you have no control over the startup times. Sure you could have heuristics such as the receiver waiting a few seconds just in case locks show up, but that just makes things slow and unpredictable. I stand by my assertion that this ca…

No, I avoid the race condition by: 1) Reader sets the lock before reading any data 2) Writer writes a byte to the pipe 3) Writer waits until pipe is empty (FIONREAD ioctl) 4) Writer checks for existance of lock.

This should be race free.

Re: Next generation Unix pipe by Alex Larsson

#79

Post by alexlarsson »

vidarh wrote: post #69
alexlarsson wrote: post #42

Earlier quoted context omitted.

I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once. And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation lik…

Your example would be about the same length with awk and sort, with the only caveat that you need to figure out the field numbers, and the upside that I can trust the tools are available pretty much everywhere.

Its doable yeah, but its a lot more work.

First you have to handle the header specially (want it in the result but not in the comparisons).

In order to compare by uid you need numeric uids (-n), but that means you can't also get the readable username, so you need a custom output format.

Then you need to ensure the output format is such that nothing with possible spaces or control chars can end up in a column before the data you're looking at, as then finding the right column is hard.

Even then, extracting the first command line arg like in the example will fail in the case of a binary name that has a space in it (as there is no way to know which spaces in the commandline corresponds to actual spaces in the arguments or just delimiters).

Re: Next generation Unix pipe by Alex Larsson

#80

Post by snprbob86 »

alexlarsson wrote: post #38
snprbob86 wrote: post #5

Neat! I've commented about this very problem before on several of the many threads regarding "object pipes", ie. REPLs. http://news.ycombinator.com/item?id=1033623 http://news.ycombinator.com/item?id=1566325 http://news.ycombinator.com/item?id=2527217 Since that last comment, I've been working a bunch with Clojure, which has a far more expressive variant of JSON, as well as some heavy duty work with Google's Protocol…

Regarding order. The dtools approach uses a stream (i.e. potentially infinite) of variants. Each variant is a self contained typed data chunk which is by itself not "streamable" (i.e. you have to read all of it). The data chunk is strongly typed and the type is self-described. The supported primitive types are: bool, byte, int16, uint16, int32, uint32, int64, uint64, double, utf8 string (+ some dbus specific things).…

Thanks. I looked at the GVariant page a bunch too.

It seems like the encoding is a steam of {type, value} pairs, where values can contain per-type headers as well.

Protobufs, on the other hand, use {field, wire-type, value} where field is required to have an externally known type to parse value, but wire-type is sufficient to determine the length of value, so you can skip unknown fields (used for backwards compatible protocols). In theory, required fields could omit field and wire-type, but Protobufs deemed it more complexity than justifies the space and performance impact.

Primitive values like integers are totally expected in any such format like this. Their salient feature being that they're of known length. I'm a little more leery about "arrays" or other data structures of variable length which are encoded with a known length. Consider Pascal strings {length, [chars]} vs C strings {[chars], NULL}. The later lends itself much better to streaming protocols, but the former is far simpler to work with when you have a complete dataset.

I ran into this situation with a Protobuf I was designing where the first attempt had a message with a repeated field, but it became obvious that I wanted a begin message, a repeated message of singular fields, and then an end message, to allow a fast-start on the send, which didn't require to know the full data set length up front.

There are, however, situations where you do want the length up front. For example, if you need to allocate space to put things. You can get faster parsing if you know the total message size immediately. In general, however, I don't think it matters all that much with modern languages and hardware.

This is one reason why Clojure has both lists and vectors. Lists are lazy head/tail pairs and (count some-vector) is a constant time operation. Unfortunately, Clojure's reader doesn't seem to offer streaming reads of lists (I may be wrong about this).

The bigger issue with unbounded values is that they are more difficult to work with in most languages. Haskell, Lisps, and other functional languages fair far better than most, but once you start mixing fixed-sized messages with known fields, with variable-sized sequences, you wind up with a situation like {x, [ys], z} where a piece of code wants to look at z before looking at ys. If that tuple is represented as an associative structure {:x 1, :ys [2 3], :z 4} then it's suddenly very confusing that it's an ORDERED map and all sorts of assumptions go out the window.

Even more fundamentally: Source code is a serialized protocol. You write down text and the order of the characters on the page have meaning. Sometimes, that order may be over-specified, but regardless, humans see order and make assumptions from it, even when order doesn't matter.

Post reply on HN