Parsing JSON in Forty Lines of Awk
1–10 of 62 posts
Re: Parsing JSON in Forty Lines of Awk
#2“except Unicode escape sequences”
Re: Parsing JSON in Forty Lines of Awk
#3Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs!
Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation.
jq and sql both have the same problem :)
Re: Parsing JSON in Forty Lines of Awk
#4Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)
$ echo "one two three four five" | awk '{$3="";print}'
one two four fiveRe: Parsing JSON in Forty Lines of Awk
#5Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)
...And once you get away from the most basic, standard set of features, the several awks in existence have diverging sets of additional features.
Re: Parsing JSON in Forty Lines of Awk
#6Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should.
Re: Parsing JSON in Forty Lines of Awk
#7Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)
> awk really shoots itself so much with its lack of features that it so desperately needs
Whence perl.
Re: Parsing JSON in Forty Lines of Awk
#8JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character
Yes, shell is definitely too weak to parse JSON!
(One reason I started https://oils.pub is because I saw that bash completion scripts try to parse bash in bash, which is an even worse idea than trying to parse JSON in bash)
I'd argue that Awk is ALSO too weak to parse JSON
The following code assumes that it will be fed valid JSON. It has some basic validation as a function of the parsing and will most likely throw an error if it encounters something strange, but there are no guarantees beyond that.
Yeah I don't like that! If you don't reject invalid input, you're not really parsing
---
OSH and YSH both have JSON built-in, and they have the hierarchical/recursive data structures you need for the common Python/JS-like API:
osh-0.33$ var d = { date: $(date --iso-8601) }
osh-0.33$ json write (d) | tee tmp.txt
{
"date": "2025-06-28"
}
$ cat tmp.txt | json read (&x)
osh-0.33$ = x
(Dict) {date: '2025-06-28'}
osh-0.33$ sed 's/"/bad/"' tmp.txt | json read (&x)
sed: -e expression #1, char 9: unknown option to `s'
sed 's/"/bad/"' tmp.txt | json read (&x)
^~~~
[ interactive ]:20: json read: Unexpected EOF while parsing JSON (line 1, offset 0-0: '')
Another example from wezm yesterday: https://mastodon.decentralised.social/@wezm/1147586026608361...
YSH has JSON natively, but for anyone interested, it would be fun to test out the language by writing a JSON parser in YSH
It's fundamentally more powerful than shell and awk because it has garbage-collected data structures - https://www.oilshell.org/blog/2024/09/gc.html
Also, OSH is now FASTER than bash, in both computation and I/O. This is despite garbage collection, and despite being written in typed Python! I hope to publish a post about these recent improvements
Re: Parsing JSON in Forty Lines of Awk
#9Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)
>awk really shoots itself so much with its lack of features that it so desperately needs!
That's why I use Perl instead (besides some short one liners in awk, which in some cases are even shorter than the Perl version) and do my JSON parsing in Perl.
This
diff -rs a/ b/ | ask '/identical/ {print $4}' | xargs rm
is one of my often used awk one liners. Unless some filenames contain e.g. whitespace, then it's Perl again
Re: Parsing JSON in Forty Lines of Awk
#10Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)
...And once you get away from the most basic, standard set of features, the several awks in existence have diverging sets of additional features.
Things are already like that, friend! We have mawk, gawk and nawk. But it's fun to think about how we could improve our ideal tooling if we had a time machine.