Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bringing the Unix philosophy to the 21st century (2019)

blog.kellybrazil.com

Post reply on HN
Print view View original

71–80 of 151 posts

Re: Bringing the Unix philosophy to the 21st century (2019)

#71

Post by dimitar »

kaliszad wrote: post #66
c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

I can see the potential in Babashka and other Clojure-based systems from the approach point of view. There needs to be a simple integrated editor or much better readline implementation though that needs to work out of the box. That should improve the interactivity in a major way. We also need to have a set of very simple tools to work with files, network connections (e.g. traffic dumps, netcat) and other typical shel…

Babashka is really nice in that it has json and yaml libraries built-in, so you end up using your Clojure data structures throughout the script, and yet still consuming/emitting json/yaml if you need to.

Re: Bringing the Unix philosophy to the 21st century (2019)

#72

Post by tuatoru »

1MachineElf wrote: post #46
c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

About this few weeks training you mentioned for Unix-style parsing of unstructured text - what resources do you recommend?

The GNU Awk manual has several practical examples in it - as manuals should. (Looking at you, nearly all man pages in linux distros.)

Re: Bringing the Unix philosophy to the 21st century (2019)

#73

Post by coliveira »

c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

You mentioned one pain point of shell scripting, variability among platforms. The other pain point is that different tools have different methods to treat errors and warnings. This makes debugging shell scripts a nightmare, compare to scripting languages like python. If you need to debug your work, you need to check several levels of script vs. commands, and use different ways to check for errors depending on what is causing the problem.

If I had to decide on a tool to replace shell script I would vote for tcl, since it maintains many of the advantages of the shell but provides a better handling of the programming aspect. Unfortunately these days it seems that you either use shell or some full-scale language like python.

Re: Bringing the Unix philosophy to the 21st century (2019)

#74

Post by azalemeth »

ajuc wrote: post #68
rurban wrote: post #50

Thankfully he didn't propose XML. Unfortunately it looks like he (and many others) really thinks that JSON is better, even if it's underspecified, thus leading to possible insecurities. See the recent jsonsec thread. (Undefined key ordering and duplicate handling) So I have to bring in jsmn.h to parse protocols? Sorry no. Been there, done that. We are pushing too much unnecessary JSON around already. Unix is also abo…

> even if it's underspecified, thus leading to possible insecurities what it replaces (unstructured text) is much less secure so I don't think that counts against json

But at least the parsers are very, very battle tested at least

Re: Bringing the Unix philosophy to the 21st century (2019)

#75

Post by kbrazil »

phendrenad2 wrote: post #70

"Up until about 2013 it made just as much sense as anything to assume unstructured text was a good way to output data at the command line..." But in 2013 a certain data format called JSON was standardized as ECMA-404..." "Had JSON been around when I was born in the 1970’s Ken Thompson and Dennis Ritchie may very well have embraced it as a recommended output format to help programs “do one thing well” in a pipeline."…

I don’t follow. Your examples are of data formats standardized in the 80’s while Unix was developed in the 60’s and 70’s. JSON even existed before 2013, but the fact that it became a standard in addition to being popular is the point I was making.

Re: Bringing the Unix philosophy to the 21st century (2019)

#76

Post by iainmerrick »

lfnoise wrote: post #42
eadmund wrote: post #23

Why not go all the way and use a format capable of expressing code and data? I refer, of course, to S-expressions. They also have the benefit of properly handling numbers. Some might look at the absence of maps as a negative, but I think alists are preferable anyway due to their constant ordering.

Yes. I never understood why JSON over s-exprs. The absence of maps is not a negative. S-exprs can represent maps. There are no maps in JSON, really anyway. It is just text. How that data is represented in memory is the output of parsing. You could just as well parse (dict (a 1)(b 2)(c 3)) into a hash table if you wanted. You could also have sets (set 1 2 3) or whatever other data structure.

You could do any of those things, but you have to pick a convention and other people have to agree on it.

JSON is nice in that it has just enough structure to do a good number of tasks in one obvious way. The biggest omission is probably some kind of time and/or date type (but ISO8601 in a string is the obvious solution there).

It’s not a coincidence that JSON was reverse-engineered from a language with convenient literals for dictionaries and arrays, and most languages provide those two collection types because they cover most use cases, so JSON fits most languages fairly well.

It’s just handy having both arrays and dictionaries available, rather than stretching one data structure to cover both, whatever Lua or Lisp might say.

Re: Bringing the Unix philosophy to the 21st century (2019)

#77

Post by anthk »

mcswell wrote: post #53

While you're at it, bring Unix into the 21st century, in which we use Unicode (UTF-8 encoding). The number of Unix utilities that badly support Unicode is, well, painful to those of us who deal with non-ASCII data all the time.

Unix "2.0" (plan9) invented utf-8.

Re: Bringing the Unix philosophy to the 21st century (2019)

#78

Post by 1vuio0pswjnm7 »

c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write.

Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If lines are too long, I can chop them down to a reasonable size on some sentinel.

IMO, JSON, like Javascript, is web/browser centric. For someone who rarely uses a browser or Javascript and is comfortable with UNIX, e.g., yours truly, JSON is not particularly advantageous. For large data, line-based is more robust (and memory-efficient) than JSON, IME.

Better than JSON is netstrings or bencode.

https://en.wikipedia.org/wiki/Netstrings

https://en.wikipedia.org/wiki/Bencode

Re: Bringing the Unix philosophy to the 21st century (2019)

#79

Post by Koshkin »

u801e wrote: post #61
JJMcJ wrote: post #54

Earlier quoted context omitted.

> awk Or Perl. That's what Perl was originally created for. Or sed. If awk is too mainstream for you. Or Python. Or just write the ugly pipeline in your script, and promptly forget how it actually works. I've done that a lot.

> Or sed After learning how to use ed in scripts, I've found it's actually easier to use it compared to sed because it effectively has random access through the input rather than going from beginning to end. But both awk and Perl have the advantage of storing values in variables over sed or ed.

sed, being a stream editor, is more efficient than ed, in that it does not need to keep the entire file in memory.

Re: Bringing the Unix philosophy to the 21st century (2019)

#80

Post by emodendroket »

c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

PowerShell is just so much nicer to use than anything where text munging is the only way to do things, and it’s just as “pluggable” as Unix shell commands. And it can output text (or JSON or XML or YAML) or whatever you want easily by piping into relevant commands (or just not piping anywhere if you wanted text). I don’t imagine those are going anywhere anytime soon but I think it is wrong to say the system cannot be improved upon.
Post reply on HN