Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

doc.cat-v.org

Post reply on HN
Print view View original

51–60 of 112 posts

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#51

Post by gits1225 »

jff wrote: post #27
gits1225 wrote: post #10

Earlier quoted context omitted.

offtopic: sam(and acme) sounds like an interesting method at first, but when you try it out, its really weird and slow to get stuff done. Its because for me, mouse is really inferior to keyboard to do the majority of the tasks. This is an except from coders at work: Seibel: Is there anything you would have done differently about learning to program? Do you have any regrets about the sort of path you took or do you wi…

continuing the offtopic: You say it sounds like an interesting method at first, but then you try it out and it's weird and slow. How much of a try did you give these editors? In the same way that a new Vim user will basically use the arrow keys, "i", ":w", and ":q" and nothing else, a new user to Sam or Acme can very easily miss a lot of the power. The Sam language is pretty powerful. For instance, you can use the "X…

I am using emacs and doing the same things in acme at the moment for around 2 weeks now. Its just that, whenever I need to type I have to disconnect from the mouse and whenever I need to do I have to disconnect from the keyboard. Also when using the mouse I miss the precision with which I can navigate to different parts of the screen real quick and efficiently. There is always a delay and re-focus to get to the target

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#52

Post by nwmcsween »

haberman wrote: post #28

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…

You're being too practical within unix constraints - metadata doesn't need to be tied to a schema or format it only needs to have metadata. Tell me what the metadata _does_ not what it is and let the protocol / format decide how to format the data. E.g. object_1 is HTTP 1.1 is ...

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#53

Post by nwmcsween »

haberman wrote: post #48
stcredzero wrote: post #36

Earlier quoted context omitted.

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix comm…

> JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dict…

Why does the protocol define the objects? Shouldn't the object metadata should have enough information to transform the output? E.g. object is a URL vs protocol is a URL

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#54

Post by wendsday »

its_so_on wrote: post #41
haberman wrote: post #28

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…

I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running an…

I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome.

The concept of lines is a human one. It is how humans parse. If humans could parse without needing the concept of a "line" then, e.g., there would be no problems with programming in C which has very shaky support for the concept of "lines".

But there are problems as we all know. It's proof that people do need to think in terms of "lines". Even though the computer does not need them. The only "problem" with this is that people are not computers.

I'm not sure anyone outside of the most unrealistic nerds would agree that this is a "problem".

1. As for the paper, I can think of at least one tool/language to work with text that does keep data in a binary format while one performs a series of transformations. It's not line-based. And I would guess that Mr. Pike does not know how to use it. It's fast and efficient. Probably faster than sam.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#55

Post by stcredzero »

dllthomas wrote: post #50
stcredzero wrote: post #39

Earlier quoted context omitted.

It could just be a set of separate tools one could pipe data to. http://news.ycombinator.com/item?id=4113231

Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. On the other hand, handling deep-enough nesting with regexp is usually not hard, and when you're stringing together a bunch of unix commands quickly you're usually looking for "good enough". I don't want to have to write a new everything to handle a new format. Maybe there's something in between?

I guess I didn't explain well enough, since you completely misunderstood my suggestion.

Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant.

Okay, thanks, but I've known about basic automata theory since I was an undergrad, two decades ago.

I had something like this in mind:

    ls -af | jsonify 'ls -af' | this_reads_a_json_stream
The jsonify command would retrieve and run a script from a central repository, which people could contribute code to. This way, the parsing efforts of one coder could be re-used by the rest of the world.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#56

Post by haberman »

wendsday wrote: post #54
its_so_on wrote: post #41

Earlier quoted context omitted.

I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running an…

I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome. The concept of lines is a human one. It is how humans parse. If humans could parse without needing the conce…

> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is?

My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern.

Let me ask you this; how would you do the equivalent of this hypothetical command?

  $ ls | structured-grep 'file.size > 1M'
The answer is that you can't in today's world without writing a parser (or some code that calls readdir/stat manually). That is the problem with unstructured text.

Text formats like CSV that seem simple actually end up being hugely complicated once you push them to their limits. Nothing is worse than software that breaks once something unexpected happens, like a string that contains an embedded comma.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#57

Post by e12e »

mey wrote: post #45
noblethrasher wrote: post #40

Earlier quoted context omitted.

Powershell on Windows does something like what you want: PS C:\Some Directory $sum = 0 PS C:\Some Directory dir | where { $_Length -gt 1MB } | %{ $sum += $_.Length }

First thing I looked for when I checked the comments, was there a mention of Powershell. Powershell has the concept of passing objects (via .net clr) instead of passing strings. It sucks when trying to deal with streams of data, but fantastic for acting as script glue between various systems. One of the things I feel Microsoft really got right.

First thing I checked for too. Second thing I looked for was someone pointing out that this is asking for object oriented systems (such as Smalltalk or Self).

But the third thing, no one mentioned: SNOBOL. Reportedly (because I haven't used it myself) it is better than AWK for complex matching.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#58

Post by bazzargh »

nhaehnle wrote: post #44
postfuturist wrote: post #42

I ran into a wall when trying to use unix-y tools to do somewhat complex, regex/replace functions for code refactors. Basically, anything inside a single line is easy, but once you cross that line barrier, the complexity increases dramatically. That rendered the changes useless because most programming languages allow you to add arbitrary new-lines between any token in the language.

I know what you're saying. I ran into the same problem a while back and ended up hacking a tool that does the kind of structured pattern matching I wanted. Its syntax is a bit awkward, but what the hell - you can find it here if you're interested: https://github.com/nhaehnle/patrex

I also hacked up a tool for structural search. I'd noticed intellij's built-in structural search has a very simple pattern language - you just type the code you're searching for and insert a wildcard like $x$ or $foo$ where an expression can vary. However, its restricted to the languages Intellij can parse.

(in what follows HN is mangling asterisks, so I used 'X')

Tokenizing a language is much simpler though. So I just did that, and let wildcards match like X? up to the next expression boundary at the same nesting level - the boundaries were ',' ';' ')' '}' and ']' when I tried this on java, C and perl.

This turns out to be simple enough and powerful enough to be useful. For instance, your example:

boost::bind(& ${id} $( :: ${id} )+, $( boost::ref( Xthis ) )|ref| $.X )

Would be: boost::bind(& $id1$ :: $id2$, boost::ref( Xthis ) ) (for 2 args - I'd need to repeat for more args like so: boost::bind(& $id1$ :: $id2$, boost::ref( Xthis ), $id3$)

This works because $id1$ is non-greedy, and whitespace is ignored. I did tend to tweak the tool to what I was searching for (I'd have dropped ',' as a delimiter for this one), which I guess is cheating on keeping the syntax simple!

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#59

Post by dllthomas »

stcredzero wrote: post #55
dllthomas wrote: post #50

Earlier quoted context omitted.

Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. On the other hand, handling deep-enough nesting with regexp is usually not hard, and when you're stringing together a bunch of unix commands quickly you're usually looking for "good enough". I don't want to have to write a new everything to handle a new format. Maybe there's something in between?

I guess I didn't explain well enough, since you completely misunderstood my suggestion. Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. Okay, thanks, but I've known about basic automata theory since I was an undergrad, two decades ago. I had something like this in mind: ls -af | jsonify 'ls -af' | this_reads_a_json_stream The jsonify command would ret…

>Okay, thanks, but I've known about basic automata theory since I was an undergrad, two decades ago.

I wasn't trying to educate; I was discussing the relevant limitations of my approach. The fact that I can't spin a perfect regexp for anything (including JSON, sexp, xml) that nests arbitrarily deeply is an issue with what I proposed - one that I think can be worked around sufficiently, but an issue nonetheless, and I wanted to acknowledge that. I don't see what prompted the defensiveness I take from your comment - I'd expect most people here to know at least that much automata theory, here. I've not known it for quite two decades, but two decades ago I was 8. I'll reply to the constructive bits of your response separately.

Post reply on HN