Given the examples in the paper and how old it is, I wonder what would be an elegant way to do it today with the available tools.
Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
101–110 of 112 posts
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#102Earlier quoted context omitted.
Or, one could write a library of standard parsers and serializers for the Unix tools that would parse and produce known JSON representations of data that could be passed between scripts.
That's recordstream
recordstream seems to concentrate on lots of formats. Really, it should concentrate on lots of tools, with a shared repository of parsers for tools.
EDIT: Actually, they concentrate on JSON, but they also try to provide a generic set of tools for reading any format. I think this could be better structured to "just work."
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#103Earlier quoted context omitted.
> The direction you're suggesting we take things is, in fact, a fuller API. Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one. > They're slower and worse. I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less…
> I'm just proposing a structured stream of data instead of an unstructured one. That's basically what an API is.
So things like CSV, XML, and JSON are APIs now?
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#104The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing t…
The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix comm…
> JSON would do nicely as well.
Until you need to incrementally process 2GB of data.
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#105Earlier quoted context omitted.
I guess I didn't explain well enough, since you completely misunderstood my suggestion. Separate tools have the advantage (over regex) of handling nesting properly, which could certainly be significant. Okay, thanks, but I've known about basic automata theory since I was an undergrad, two decades ago. I had something like this in mind: ls -af | jsonify 'ls -af' | this_reads_a_json_stream The jsonify command would ret…
It appears that I did misunderstand your suggestion - I'd thought you to be proposing a separate set of tools (grep, sed, etc) for operating on each packet format. Instead, it appears you meant a set of tools for converting output to a particular format? That still means either 1) reimplementing tools to deal with each format, or 2) many conversions and perhaps still an inability to get the data to chunk the desired…
If one could simply get all of the data parsed to a particular format, people using such tools could easily pick which data they need and discard the rest.
Also, there would be "many conversions" - but I'm envisioning that these would be shared in a library, so it would "just work" for most developers.
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#106Earlier quoted context omitted.
> The direction you're suggesting we take things is, in fact, a fuller API. Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one. > They're slower and worse. I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less…
> Communicating with a person is an endless process of content negotiation. What are you and I talking about? Right. So how does grep know it's talking to ls about dir contents with a (disk) size field and not talking to ps about process table contents with a (mem) size field? Some sort of slower content negotiation, I presume. > cannot possibly I suggest that every time you think that, you double-check your assumpti…
His point is that he's pushing off the extra hard work to be done only once, rather than at every pipe. I'm not sure how he needs to recheck his assumptions there since he stated them.
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#107Earlier quoted context omitted.
The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix comm…
> JSON would do nicely as well. Until you need to incrementally process 2GB of data.
BSON in that case?
Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#108Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#109Re: Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"
#110Earlier quoted context omitted.
> I'm just proposing a structured stream of data instead of an unstructured one. That's basically what an API is.
You are confusing the "how" (the API specification) with the "what" (the data the API exists to provide access to).
> I'm experimenting with the idea of letting people specify the output of command-line utilities as a Protocol Buffer schema, for example:
Looks like an API spec to me