Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bringing the Unix philosophy to the 21st century (2019)

blog.kellybrazil.com

Post reply on HN
Print view View original

91–100 of 151 posts

Re: Bringing the Unix philosophy to the 21st century (2019)

#91

Post by 1vuio0pswjnm7 »

user-the-name wrote: post #89
1vuio0pswjnm7 wrote: post #78

Earlier quoted context omitted.

Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write. Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If…

> JSON's design assumes the user can read the entire file into memory It doesn't. I have personally written JSON parsing code that runs on embedded systems with less RAM than the size of the file.

"I have personally written JSON parsing code that runs on embedded systems with less RAM than the size of the file."

I have too. I used shell utilities. :)

YAJL will also work.

Re: Bringing the Unix philosophy to the 21st century (2019)

#92

Post by emodendroket »

diegocg wrote: post #88
emodendroket wrote: post #80

Earlier quoted context omitted.

PowerShell is just so much nicer to use than anything where text munging is the only way to do things, and it’s just as “pluggable” as Unix shell commands. And it can output text (or JSON or XML or YAML) or whatever you want easily by piping into relevant commands (or just not piping anywhere if you wanted text). I don’t imagine those are going anywhere anytime soon but I think it is wrong to say the system cannot be…

I had to write a powershell script and my impression was that objects are a much worse interface than the worst text manipulation tricks. Because you are passing objects around it quickly becomes really hard to understand what is going on on the code and which object interfaces the script is using, because the object lives behind the pipe so to speak. It was my impression that it is too easy to create unreadable powe…

I'm not sure I understand exactly what you mean. I would rather read Select { $_.Property } than some awk or sed stuff to fetch the same thing any day. I'm not sure how this is different than most other modern languages used to write scripts.

Re: Bringing the Unix philosophy to the 21st century (2019)

#93

Post by mananaysiempre »

1vuio0pswjnm7 wrote: post #90
mananaysiempre wrote: post #85

Earlier quoted context omitted.

> JSON’s design assumes the user can read the entire file into memory No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces…

All three of those Python scripts use the same library, YAJL It comes with an example program called json_reformat which I have experimented with in the past. However the "reformatters" I write using only shell utilities work just as well. YMMV.

Yes, but raw YAJL (or its Yajl-Py binding, or other incremental parsers like jsmn) is a right pain to use while these are actually interesting in terms of API design.

I doubt your JSON reformatter was entirely correct (hey, we all must parse [X]HTML using regex from time to time), as you either have to pretend you can make sense of JSON using regular expressions (you can’t, no language or format supporting unbounded nesting can be regular) or write what’s essentially a standard JSON parser in shell with all the associated inefficiencies. My own needs are usually adequately served by the likes of jq -r ... | while IFS=$"\t" read -r ..., which feels much less hackish.

(Even correctly handling CSV with quoting, escaping, and embedded newlines in a UNIX pipeline without a purpose-built utility is surprisingly difficult—although not impossible, as CSV is a regular language.)

Re: Bringing the Unix philosophy to the 21st century (2019)

#94

Post by lfnoise »

iainmerrick wrote: post #76
lfnoise wrote: post #42

Earlier quoted context omitted.

Yes. I never understood why JSON over s-exprs. The absence of maps is not a negative. S-exprs can represent maps. There are no maps in JSON, really anyway. It is just text. How that data is represented in memory is the output of parsing. You could just as well parse (dict (a 1)(b 2)(c 3)) into a hash table if you wanted. You could also have sets (set 1 2 3) or whatever other data structure.

You could do any of those things, but you have to pick a convention and other people have to agree on it. JSON is nice in that it has just enough structure to do a good number of tasks in one obvious way. The biggest omission is probably some kind of time and/or date type (but ISO8601 in a string is the obvious solution there). It’s not a coincidence that JSON was reverse-engineered from a language with convenient li…

You parse the s-exprs and execute them in the context of a namespace of data constructors. Then you can have whatever data structures in memory that are defined by the constructors. This is NOT equivalent to having one data structure to cover both as Lua does. It is having one text format that can construct any kind of data structure in memory for which you have constructors defined.

Re: Bringing the Unix philosophy to the 21st century (2019)

#95

Post by 1vuio0pswjnm7 »

mananaysiempre wrote: post #93
1vuio0pswjnm7 wrote: post #90

Earlier quoted context omitted.

All three of those Python scripts use the same library, YAJL It comes with an example program called json_reformat which I have experimented with in the past. However the "reformatters" I write using only shell utilities work just as well. YMMV.

Yes, but raw YAJL (or its Yajl-Py binding, or other incremental parsers like jsmn) is a right pain to use while these are actually interesting in terms of API design. I doubt your JSON reformatter was entirely correct (hey, we all must parse [X]HTML using regex from time to time), as you either have to pretend you can make sense of JSON using regular expressions (you can’t, no language or format supporting unbounded…

Thanks for the summary of problems with JSON. For ease of use, it will not likely become a de facto interchange format in the UNIX context. However not all JSON is the same. Simple JSON is easy to parse and one does not need jq or libraries, for example JSON DoH responses. People have even tried to standardise simpler JSON, e.g., jsonl, as another reply mentions. But, as I said, the design of JSON allows for and encourages complexity. If that's what you like, go for it. An HN commenter recently identified the "complexity fetish" that many programmers have.

Not everyone suffers from that fetish, fortunately. With excessively complex JSON, I prefer to extract the specific data I want. The speed of json_reformat demonstrates how slow reformatting is when done "correctly".

Every user's needs are different. jq fails to meet mine.

Re: Bringing the Unix philosophy to the 21st century (2019)

#96

Post by u801e »

Koshkin wrote: post #79
u801e wrote: post #61

Earlier quoted context omitted.

> Or sed After learning how to use ed in scripts, I've found it's actually easier to use it compared to sed because it effectively has random access through the input rather than going from beginning to end. But both awk and Perl have the advantage of storing values in variables over sed or ed.

sed, being a stream editor, is more efficient than ed, in that it does not need to keep the entire file in memory.

True, but if the file isn't that big, then the difference isn't noticable.

Being able to do something like

  /some string/-2 d
Which deletes a line 2 lines above where some string matched is something that's trivial to do in ed, but takes a bit of work in sed.

Re: Bringing the Unix philosophy to the 21st century (2019)

#97

Post by guenthert »

zozbot234 wrote: post #12
natt941 wrote: post #5

I can understand why the idea of more structured, object-like input and output is appealing, but after using PowerShell for a while, my take is that it's much harder to manipulate objects into a consistent format than it is to manipulate text. For instance, if you want to compare Azure DNS records with DNS records from a Windows server, it's a huge pain because Get-AzDnsRecordSet and Get-DnsServerResourceRecord retur…

> More generally, text is great for loose coupling; structured objects, less so. I disagree. Parsing "loose coupled" text and converting it to the format that a different tool expects is a rather non-trivial problem, and one that's often poorly specified to begin with; converting structured outputs is generally straightforward in comparison.

Sure, converting (or interpreting) structured data is easier then unstructured. I doubt anyone is arguing that. I interpret op's statement however in that unstructured data is easier to couple unrelated tools. Tools perhaps which haven't written yet, by teams not knowing of each other and hence are unable to agree on a structure.

Re: Bringing the Unix philosophy to the 21st century (2019)

#98

Post by rodgerd »

kbrazil wrote: post #21
c0l0 wrote: post #19

So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…

I don’t think the issue is that it’s hard to manually parse. The problem is that it’s hard for someone else to read your ad-hoc parser years later and reason about what you did if they need to modify it. Disclaimer: I am the author of the article and JC.

But it's very gratifying to the person who re-invents the wheel for the millionth time and feels good because of it.

The Unix command line is the Candy Crush of interfaces, giving us a little dopamine hit every time we solve a problem that didn't need to be solved.

Re: Bringing the Unix philosophy to the 21st century (2019)

#99

Post by lakecresva »

bassman9000 wrote: post #11
richardanaya wrote: post #2

I was literally just thinking about this a few days ago. I'm super excited by https://www.nushell.sh/ . I think they are hitting on an order of magnitude improvement paradigm of shells that fit very nicely with the theme of this article.

https://www.nushell.sh/book/ This goes against the philosophy explicitly mentioned in OP's article. E.g. avoid tabular formats. This is systemd against init again. A powerful, but overreaching shell that becomes unreplaceable and bloated with concerns. In constrast, traditional *nix/GNU programs work well, and interact well, in every shell.

Nushell has a 'to json' command. The tables are just nice to look at, it's not a data format they're expecting people to parse.

Re: Bringing the Unix philosophy to the 21st century (2019)

#100

Post by chillfox »

yesenadam wrote: post #87
1MachineElf wrote: post #46

Earlier quoted context omitted.

About this few weeks training you mentioned for Unix-style parsing of unstructured text - what resources do you recommend?

The original 1988 book The AWK Programming Language by A, W and K is, in my opinion, one of the finest pieces of documentation ever written, on any subject. A joy to read and be instructed by.

And $722 AUD on Amazon. Did they never make a second printing?
Post reply on HN