Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

doc.cat-v.org

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#61

Post by dllthomas »

stcredzero wrote: post #55
dllthomas wrote: post #50

Earlier quoted context omitted.

Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. On the other hand, handling deep-enough nesting with regexp is usually not hard, and when you're stringing together a bunch of unix commands quickly you're usually looking for "good enough". I don't want to have to write a new everything to handle a new format. Maybe there's something in between?

I guess I didn't explain well enough, since you completely misunderstood my suggestion. Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. Okay, thanks, but I've known about basic automata theory since I was an undergrad, two decades ago. I had something like this in mind: ls -af | jsonify 'ls -af' | this_reads_a_json_stream The jsonify command would ret…

It appears that I did misunderstand your suggestion - I'd thought you to be proposing a separate set of tools (grep, sed, etc) for operating on each packet format. Instead, it appears you meant a set of tools for converting output to a particular format? That still means either 1) reimplementing tools to deal with each format, or 2) many conversions and perhaps still an inability to get the data to chunk the desired way for any particular tool. Just the ability to have sed, grep, and sort chunk in an arbitrary way would be significant.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#62

Post by neutronicus »

haberman wrote: post #48
stcredzero wrote: post #36

Earlier quoted context omitted.

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix comm…

> JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dict…

How can your tools all accept the same kind of structured data without dictating its representation? I don't get it.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#63

Post by ori_b »

e12e wrote: post #57
mey wrote: post #45

Earlier quoted context omitted.

First thing I looked for when I checked the comments, was there a mention of Powershell. Powershell has the concept of passing objects (via .net clr) instead of passing strings. It sucks when trying to deal with streams of data, but fantastic for acting as script glue between various systems. One of the things I feel Microsoft really got right.

First thing I checked for too. Second thing I looked for was someone pointing out that this is asking for object oriented systems (such as Smalltalk or Self). But the third thing, no one mentioned: SNOBOL. Reportedly (because I haven't used it myself) it is better than AWK for complex matching.

I played with SNOBOL. It may have a more powerful matching engine, but power doesn't equate to usability. Having each line followed by 3 gotos does not good UX make.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#64

Post by e12e »

haberman wrote: post #56
wendsday wrote: post #54

Earlier quoted context omitted.

I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome. The concept of lines is a human one. It is how humans parse. If humans could parse without needing the conce…

> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern. Let me ask you this; how would you do the equivalent of this hypothetical command? $ ls | structured-grep 'file.size > 1M' The answer is that you can't in today's world without wri…

"ls" prints a list of files. "find" knows about the filesystem and can print a list of files based on file metadata and a path (such as a filename). If you have a list of files (eg from "ls") -- you would need to look up the metadata you want to filter on; it is not part of ls' interface to give them to you directly:

  ls | find -size +1M
On the other hand, if you want to make list of files and their sizes, that can be stored, sent over the network, etc -- and then filer that list, you can do:

  ls --size | awk 'file_blocks=$1 $file_blocks > 1024 { print $2 }'
For most tasks, I think the fact that any human can look at the output from eg: "ls --size" and then produce a valid test dataset in the same format is more valuable than having to explicitly "cast" the metadata while processing.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#65

Post by swa14 »

haberman wrote: post #56
wendsday wrote: post #54

Earlier quoted context omitted.

I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome. The concept of lines is a human one. It is how humans parse. If humans could parse without needing the conce…

> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern. Let me ask you this; how would you do the equivalent of this hypothetical command? $ ls | structured-grep 'file.size > 1M' The answer is that you can't in today's world without wri…

>>Let me ask you this; how would you do the equivalent of this hypothetical command? >> $ ls | structured-grep 'file.size > 1M'

# find . -maxdepth 1 -size +1M

really, flat text is fine. Maybe it's not perfect, but it's good enough that most want to not add any complexity to it that would make it incompatible.

And if you really need that complexity, it's usually worth whipping up a parser for.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#66

Post by jcmhn »

neutronicus wrote: post #62
haberman wrote: post #48

Earlier quoted context omitted.

> JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dict…

How can your tools all accept the same kind of structured data without dictating its representation? I don't get it.

Allow a shell variable to control output record and field separators. Default to space and newline if nothing is specified.

ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,...

Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#67

Post by akkartik »

saraid216 wrote: post #60
silas wrote: post #16

Every scribd link is marked as private for me Hacker News for some reason, is this broken for anyone else or..?

Me, too. I'd really like to read this...

But surely the main non-scribd link works?

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#68

Post by haberman »

swa14 wrote: post #65
haberman wrote: post #56

Earlier quoted context omitted.

> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern. Let me ask you this; how would you do the equivalent of this hypothetical command? $ ls | structured-grep 'file.size > 1M' The answer is that you can't in today's world without wri…

>>Let me ask you this; how would you do the equivalent of this hypothetical command? >> $ ls | structured-grep 'file.size > 1M' # find . -maxdepth 1 -size +1M really, flat text is fine. Maybe it's not perfect, but it's good enough that most want to not add any complexity to it that would make it incompatible. And if you really need that complexity, it's usually worth whipping up a parser for.

"find" is a poor man's "structured-grep." It provides a bunch of functionality for filtering a result set, but is totally specific to lists of files. You can't use find with ps, netstat, iptables, ifconfig, or any other command-line program that produces a list of records.

> And if you really need that complexity, it's usually worth whipping up a parser for.

No work is worth doing if it could just as easily have been avoided.

The vision in my head has less complexity than the status quo, not more. How many flags does "find" have? One for every field name you can filter/sort by. Done right, a "structured-grep" that can grep on any field sent to it is much, much simpler.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#69

Post by haberman »

neutronicus wrote: post #62
haberman wrote: post #48

Earlier quoted context omitted.

> JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dict…

How can your tools all accept the same kind of structured data without dictating its representation? I don't get it.

When I talk about a "representation," I mean an in-memory format. For example, the "representation" of an HTML tree is the DOM.

Yes, you have to agree on a serialization format (JSON, Protocol Buffers, etc), but that's not the same thing. From a serialization format you can represent the data however you see fit in your process. For example, a C++ user might represent a string as a std::string object whereas a Python user would represent it as a native Python string.

The VM-based approach (like PowerShell) defines an in-memory tree representation, namely .NET objects. This means that you can't really interoperate with this stack unless you use .NET too, since you don't have an easy way of converting .NET objects to your own objects.

Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

#70

Post by pwpwp »

haberman wrote: post #28

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…

Problem with PBs is that the receiver needs a schema to parse. JSON or S-expressions obviate that need.
Post reply on HN