Next generation Unix pipe by Alex Larsson
blogs.gnome.org
Next generation Unix pipe by Alex Larsson
1–10 of 86 posts
Re: Next generation Unix pipe by Alex Larsson
#2I like the idea of processes dumping structured objects: pipes are rather often used for the processing of structured data, and while tabulated output certainly makes it easier, we still end up effectively using constants: cut to the third column, sort the first 10 characters, and print the first four lines.
This method is fragile when given diverse input: what if the columns could themselves contain tabs, newlines, or even nul bytes?
Passing objects as binary blobs, on the other hand, doesn't allow for ease of display or interoperability with other tools that don't support whatever format they happen to be. This, of course, can be rectified with a smart shell with pretty-print for columnar data (insofar as a shell could be charged with data parsing; you may imagine an implicit |dprint at the end of each command line that outputs blobs).
I'd also be interested in seeing a utility that took "old-format" columnar data and generated structured objects from it, of course, with the above format caveats.
Re: Next generation Unix pipe by Alex Larsson
#3If I can do a slight PG impression, "what problem does this solve?"
Re: Next generation Unix pipe by Alex Larsson
#4If I can do a slight PG impression, "what problem does this solve?"
Don't need to always use sed and awk. You can more easily sort, without recoursing to cut. You can let your pipe work on ranges.
Basically, use your imagination :-)
Re: Next generation Unix pipe by Alex Larsson
#5Neat! I've commented about this very problem before on several of the many threads regarding "object pipes", ie. REPLs.
http://news.ycombinator.com/item?id=1033623
http://news.ycombinator.com/item?id=1566325
http://news.ycombinator.com/item?id=2527217
Since that last comment, I've been working a bunch with Clojure, which has a far more expressive variant of JSON, as well as some heavy duty work with Google's Protocol Buffers.
A few points:
1) Piping non-serializable objects is a BAD IDEA. That's not a shell, that's a REPL. And even in a REPL, you should prefer inert data, a la Clojure's immutable data structures.
2) Arbitrary bit streams is, fundamentally, unbeatable. It's completely universal. Some use cases really don't want structured data. Consider gzip: you just want to take bytes in and send bytes out. You don't necessarily want typed data in the pipes, you want typed pipes, which may or may not contain typed data. This is the "format negotiation" piece that is mentioned in the original post. I'd like to see more details about that.
3) There seems to be some nebulous lowest common denominator of serializable data. So many things out there: GVariant, Clojure forms, JSON, XML, ProtoBuf, Thirft, Avro, ad infinitum. If everything talks its own serialization protocol, then none of the "do one thing well" benefits work. Every component needs to know every protocol. One has to "win" in a collaborative shell environment. I need to study GVariant more closely.
4) Whichever format "wins", it needs to be self-describing. A table format command can't work on field names, unless it has the field names! ProtoBufs and Thrift are out, because you need to have field names pre-compiled on either side of the pipe. Unless, of course, you start with a MessageDescriptor object up front, which ProtoBufs support and Avro has natively, but I digress: Reflection is necessary. It's not clear if you need header descriptors a la MessageDescriptor/Avro, or inline field descriptions a la JSON/XML/Clojure. Or a mix of both?
5) Order is critical. There's a reason these formats are called "serializable". Clojure, for example, provides sets using the #{} notation. And, like JSON, supports {} map notation. Thrift has Maps and Sets too. ProtoBufs, however, don't. On purpose. And it's a good thing! The data is going to come across the pipe in series, so a map or set doesn't make sense. Use a sequence of key-value-pairs. It might even be an infinite sequence! It's one thing to support un-ordered data when printing and reading data. It's another thing entirely to design a streaming protocol around un-ordered data. Shells need a streaming protocol.
6) Going back to content negotiation, this streaming protocol might be able to multiplex types over a single stream. Maybe gzip sends a little structured metadata up front, then a binary stream. ProtoBufs label all "bytes" fields with a size, but you might not know the size in advanced. Maybe you need two synchronized streams on which you can multiplex a control channel? That is, each pipe is two pipes. One request/response pair and the other a modal byte stream vs typed message stream.
Overall. This is the nicest attempt at this idea I've seen yet. I've been meaning to take a crack at it myself, but refused to do it without enough time to re-create the entire standard Unix toolkit plus my own shell ;-)
Re: Next generation Unix pipe by Alex Larsson
#6I like the idea of processes dumping structured objects: pipes are rather often used for the processing of structured data, and while tabulated output certainly makes it easier, we still end up effectively using constants: cut to the third column, sort the first 10 characters, and print the first four lines. This method is fragile when given diverse input: what if the columns could themselves contain tabs, newlines,…
Something like a cut, only we call it dcut? Actually sounds like a pretty good idea - that way those who don't want to switch to the new format don't have to, and you can pipe it through this program to create the new style structured output...
Re: Next generation Unix pipe by Alex Larsson
#7What about providing a filter that converts to whatever format you can think of? e.g. outputs in JSON or XML
Re: Next generation Unix pipe by Alex Larsson
#8If I can do a slight PG impression, "what problem does this solve?"
Don't need to always use sed and awk. You can more easily sort, without recoursing to cut. You can let your pipe work on ranges. Basically, use your imagination :-)
What's wrong with sed and awk? This may be the Stockholm Syndrome talking, but I like awk!
As for the sorting, what's wrong with using the -k flag? I only use cut when I really don't care about that field.
Re: Next generation Unix pipe by Alex Larsson
#9If I can do a slight PG impression, "what problem does this solve?"
Don't need to always use sed and awk. You can more easily sort, without recoursing to cut. You can let your pipe work on ranges. Basically, use your imagination :-)
Well I'm not going to learn a new tool to solve imaginary problems. Is there anything in the real world, a simple scenario where this would be handier than tools that have existed for 20+ years?
Re: Next generation Unix pipe by Alex Larsson
#10I like the idea of processes dumping structured objects: pipes are rather often used for the processing of structured data, and while tabulated output certainly makes it easier, we still end up effectively using constants: cut to the third column, sort the first 10 characters, and print the first four lines. This method is fragile when given diverse input: what if the columns could themselves contain tabs, newlines,…
How would a column contain a newline?