Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

doc.cat-v.org

Post reply on HN
Print view View original

41–50 of 112 posts

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#41

Post by its_so_on »

haberman wrote: post #28

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…

I like your observation but I don't like your conclusion.

I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running analysis to Twitter to announce that you've finished computing it, no matter how many switches you add, because twitter is not a commandline app.

The direction you're suggesting we take things is, in fact, a fuller API. These exist. They're slower and worse.

The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person.

Communication is faster with a mind-meld where you're looking at another person's face and picking up micro-expressions and body language. That's also the easiest thing to misinterpret.

When you pipe through such a 'human-readable' lowest common denominator, you're actually setting the ground for a very dynamic and versatile channel.

I don't know what the easy solution is to the problem you bring up, but I don't think your proposal is it.

Maybe there is no easy solution. Several unicode characters to abstract away the tab character and the newline character to instead n dimensions of characters, and corresponding negative characters so you can put something on a 'line' out of channel, (negative tab comment, tab, text, newline repeat) would probably solve some issues but is too abstract to even discuss. What we have really isn't that bad.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#42

Post by postfuturist »

I ran into a wall when trying to use unix-y tools to do somewhat complex, regex/replace functions for code refactors. Basically, anything inside a single line is easy, but once you cross that line barrier, the complexity increases dramatically. That rendered the changes useless because most programming languages allow you to add arbitrary new-lines between any token in the language.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#43

Post by zem »

stcredzero wrote: post #38
anvandare wrote: post #33

Earlier quoted context omitted.

Actually I've been considering/dreaming about a thing like this for a long while now (ever since I got started with jQuery and discovered how smooth I could sail through the DOM with it). The idea I came up with (I'm most likely not the first to think of it, so tell me if someone already implemented it) is a sort of JavaScript shell for *nix that'd work similar to jQuery: passing along (collections of) JavaScript obj…

Or, one could write a library of standard parsers and serializers for the Unix tools that would parse and produce known JSON representations of data that could be passed between scripts.

i did something like that at my last job - the cluster manager was designed as a set of command line tools, and every program had a -j flag that would make it parse stdin as json and write json to stdout. we then wrote wrappers around all the linux utilities we were using, to enable the same behaviour. once the basic system was in place it let us experiment with new features very rapidly indeed because everything could be tested in isolation from the command line using automated test scripts. what was especially nice was that we could use the same test framework to chain several utilities together and test the combination, because it was all a black box with json going in and coming out.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#44

Post by nhaehnle »

postfuturist wrote: post #42

I ran into a wall when trying to use unix-y tools to do somewhat complex, regex/replace functions for code refactors. Basically, anything inside a single line is easy, but once you cross that line barrier, the complexity increases dramatically. That rendered the changes useless because most programming languages allow you to add arbitrary new-lines between any token in the language.

I know what you're saying. I ran into the same problem a while back and ended up hacking a tool that does the kind of structured pattern matching I wanted. Its syntax is a bit awkward, but what the hell - you can find it here if you're interested: https://github.com/nhaehnle/patrex

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#45

Post by mey »

noblethrasher wrote: post #40
anvandare wrote: post #33

Earlier quoted context omitted.

Actually I've been considering/dreaming about a thing like this for a long while now (ever since I got started with jQuery and discovered how smooth I could sail through the DOM with it). The idea I came up with (I'm most likely not the first to think of it, so tell me if someone already implemented it) is a sort of JavaScript shell for *nix that'd work similar to jQuery: passing along (collections of) JavaScript obj…

Powershell on Windows does something like what you want: PS C:\Some Directory $sum = 0 PS C:\Some Directory dir | where { $_Length -gt 1MB } | %{ $sum += $_.Length }

First thing I looked for when I checked the comments, was there a mention of Powershell. Powershell has the concept of passing objects (via .net clr) instead of passing strings. It sucks when trying to deal with streams of data, but fantastic for acting as script glue between various systems.

One of the things I feel Microsoft really got right.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#46

Post by buster »

I guess one can say that 25 years later this has been shown as (partly) wrong.

I don't think Powershell really solves a problem, it's too complex to work with for the majority of problems. If i want complex data handling i write a script and put #!/usr/bin/env {bash,python,perl} in the first line.

I think the missing point here is that the nice, line-based, really simple approach is that this is how we speak and write and think. (in a series of flat words, so to speak). It's extremely easy to get into this kind of handling "data" and it's sufficient for a lot of tasks. I always admired what can be done with one line of bash/GNU utils.

As i said: If it get's more complex, we use a "real" programming language with more complex data structures anyway.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#47

Post by haberman »

its_so_on wrote: post #41
haberman wrote: post #28

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…

I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running an…

> The direction you're suggesting we take things is, in fact, a fuller API.

Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one.

> They're slower and worse.

I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less work cannot possibly be slower.

> The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person.

Communicating with a person is an endless process of content negotiation. What are you and I talking about? Using our interface of plain text, we could be talking about literally anything. I could suddenly start talking about kumquat farming in Russia. At that point you could decide to follow suit and weigh in with your opinions on kumquat farming, or you could stop talking to me altogether because I've gone off-topic. If I start speaking complete jibberish, you could try to learn the language that I'm speaking, or you could start doing something equally nonsensical.

You and I can respond to unexpected communications in useful ways because we are fully autonomous, intelligent, sovereign beings that are capable of learning, creativity, and curiosity. I don't know about you, but I don't want my software to be autonomous or react to unexpected situations in unpredictable ways.

Data processing software should be as simple, predictable, and deterministic as possible. To use your example, if I somehow got an email address in my list called "send $1000 to Lucy," I don't want PayPal to decide to get smart and interpret the invalid email address as a command to send money to Lucy.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#48

Post by haberman »

stcredzero wrote: post #36
haberman wrote: post #28

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix comm…

> JSON would do nicely as well.

Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream

> Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.

Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dictating representations of objects, you are dictating far too much about the implementation of individual pipeline nodes.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#49

Post by imd »

dougabug wrote: post #25
sitkack wrote: post #18

His example is poor, but the message is that the body of a record can cross line boundaries. While the UNIX tool chain is predicated on the concept line == record, this doesn't have to be the case. With a generic record level marshalling system the class of problems solved by composing command line tools together would be greatly expanded. What Pike describes is analogous to the RecordReader in Hadoop.

Except that Pike's paper precedes Hadoop by two decades.

That's not an 'except', grandparent never implied otherwise.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#50

Post by dllthomas »

stcredzero wrote: post #39
dllthomas wrote: post #23

In terms of flexibility, I think this would be a fantastic addition to the tools. Having it be a shell var instead of an argument might be worthwhile - if I have a few stages in a pipeline dealing with the same kind of record, it seems useful to be able to say ( RECORD_PATTERN=somepattern; my | pipe | line | whatever ) rather than my -R somepattern | pipe -R somepattern | line -K somepattern | whatever -R somepattern

It could just be a set of separate tools one could pipe data to. http://news.ycombinator.com/item?id=4113231

Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. On the other hand, handling deep-enough nesting with regexp is usually not hard, and when you're stringing together a bunch of unix commands quickly you're usually looking for "good enough". I don't want to have to write a new everything to handle a new format. Maybe there's something in between?
Post reply on HN