JSON is a terrible intermediary format since its structure is incompatible with streaming data; and demands full structure parsing before consuming the data.
Bringing the Unix philosophy to the 21st century (2019)
81–90 of 151 posts
Re: Bringing the Unix philosophy to the 21st century (2019)
#82Earlier quoted context omitted.
This is even more true for the ungodly long `jq` incantations that people write. It's like I get it, the old way is ugly and not always easy to decipher but at least it's shorter and your chances of understanding it are better. I've had both -- the classic piped chain of UNIX commands and various JSON-producing tools piped to `jq`. The former were still easier to work with.
(Sorry, this is my first time trying to do a formatted comment here.) What I like to do is comments like: /* * Collates * [ * { * id: 4 * dept: 'oncology', * name: 'Joe S.' * } * . * . * . * ] * * into * * { * 4: { // id * 'Joe. S': { // name * dept: oncology * } * . * . * . * } */ ( Then insert horrible one-liner that does the transformation. )
Two spaces in front of a line will create a code block:
/*
* Collates
* [
...
Re: Bringing the Unix philosophy to the 21st century (2019)
#83JSON is a terrible intermediary format since its structure is incompatible with streaming data; and demands full structure parsing before consuming the data.
The input and output of jq are JSON streams:
> jq filters run on a stream of JSON data. The input to jq is parsed as a sequence of whitespace-separated JSON values which are passed through the provided filter one at a time. The output(s) of the filter are written to standard out, again as a sequence of whitespace-separated JSON data.
Re: Bringing the Unix philosophy to the 21st century (2019)
#84So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…
Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write. Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If…
JSON Lines is often used for this type of thing.
Re: Bringing the Unix philosophy to the 21st century (2019)
#85So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…
Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write. Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If…
> JSON’s design assumes the user can read the entire file into memory
No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces of data that are nested more than one level down.
The lack of length prefixes in JSON does indeed make a solid parser somewhat more difficult, but you get the ability to author and validate it manually instead. All in all a draw and not because of the incremental parsing thing.
(Tabular or otherwise homogeneous data is indeed reprsented wastefully, but unless the individual records are huge json+gzip is a perfectly serviceable “worse-is-better” solution, and its self-describing nature along with support for structured cell values can at times make for a better experience than TSV. And at other times not.)
[1] https://github.com/ICRAR/ijson
Re: Bringing the Unix philosophy to the 21st century (2019)
#86I can understand why the idea of more structured, object-like input and output is appealing, but after using PowerShell for a while, my take is that it's much harder to manipulate objects into a consistent format than it is to manipulate text. For instance, if you want to compare Azure DNS records with DNS records from a Windows server, it's a huge pain because Get-AzDnsRecordSet and Get-DnsServerResourceRecord retur…
There are better shells out there for handling structural data, like Murex and Elvish
No seriously, there really are. I appreciate it requires learning a new tool but rather than downvoting me how about those unconvinced ask me questions instead? I’m happy to answer.
While there will always be a need to keep Bash around for comparability, there are a plethora of other tools out there that solve many of the shortcomings of POSIX.
Re: Bringing the Unix philosophy to the 21st century (2019)
#87So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…
About this few weeks training you mentioned for Unix-style parsing of unstructured text - what resources do you recommend?
The original 1988 book The AWK Programming Language by A, W and K is, in my opinion, one of the finest pieces of documentation ever written, on any subject. A joy to read and be instructed by.
Re: Bringing the Unix philosophy to the 21st century (2019)
#88So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…
PowerShell is just so much nicer to use than anything where text munging is the only way to do things, and it’s just as “pluggable” as Unix shell commands. And it can output text (or JSON or XML or YAML) or whatever you want easily by piping into relevant commands (or just not piping anywhere if you wanted text). I don’t imagine those are going anywhere anytime soon but I think it is wrong to say the system cannot be…
I had to write a powershell script and my impression was that objects are a much worse interface than the worst text manipulation tricks. Because you are passing objects around it quickly becomes really hard to understand what is going on on the code and which object interfaces the script is using, because the object lives behind the pipe so to speak. It was my impression that it is too easy to create unreadable powershell scripts
It also was unbearably slow in a modern computer, to the point that it made debugging difficult.
Re: Bringing the Unix philosophy to the 21st century (2019)
#89So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise. Having JSON…
Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write. Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If…
> JSON's design assumes the user can read the entire file into memory
It doesn't. I have personally written JSON parsing code that runs on embedded systems with less RAM than the size of the file.
Re: Bringing the Unix philosophy to the 21st century (2019)
#90Earlier quoted context omitted.
Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write. Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If…
> JSON’s design assumes the user can read the entire file into memory No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces…
All three of those Python scripts use the same library, YAJL
It comes with an example program called json_reformat which I have experimented with in the past.
However the "reformatters" I write using only shell utilities work just as well. YMMV.