Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON in Forty Lines of Awk

akr.am

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Parsing JSON in Forty Lines of Awk

#31

Post by toddm »

chaps wrote: post #3

Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)

Us old UNIX guys would likely go for cut for this sort of task:

     cut -d " " -f1-2,4-5 file.txt
where file.txt is:

     one two three four five
and the return is:

     one two four five

Re: Parsing JSON in Forty Lines of Awk

#32

Post by heresie-dabord »

SoftTalker wrote: post #7
chaps wrote: post #3

Awk is great and this is a great post. But dang, awk really shoots itself so much with its lack of features that it so desperately needs! Like: printing all but one column somewhere in the middle. It turns into long, long commands that really pull away from the spirit of fast fabrication unix experimentation. jq and sql both have the same problem :)

> awk really shoots itself so much with its lack of features that it so desperately needs Whence perl.

Using the Perl module JSON::PP, the programme json_pp is installed.

The following command is handy for grepping the output:

    cat mydata.json  | json_pp

Re: Parsing JSON in Forty Lines of Awk

#33

Post by chaps »

SoftTalker wrote: post #28
chaps wrote: post #27

Earlier quoted context omitted.

Kinda, but not really. Of the infrastructures I've worked on, not a single one has been consistent in installing perl on 100% of hosts. The ones that get close are usually like that because one high up person really, really likes perl. And they send a lot of angry emails about perl not being installed. Within infrastructures where perl is installed on 95% of hosts, that 5% really bites you in the ass and leads to inf…

Same with Python. It's mostly available, but sometimes not. With Perl, I find that a base installation is almost always available, but many packages might not be.

I dunno about that. IME, python is much, much more universally installed on the hosts I've worked on. Sure, usually it's 2.7, but it's there! I've tended to work on rhel and debian hosts, with some fedora in the mix.

(Once had a coworker reject a PR I wrote because I included a bash builtin in a deployment script. He said that python is more likely to be installed than bash, so we should not use bash. These debates are funny sometimes.)

Re: Parsing JSON in Forty Lines of Awk

#34

Post by comex »

packetlost wrote: post #11
chubot wrote: post #8

JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character Yes, shell is definitely too weak to parse JSON! (One reason I started https://oils.pub is because I saw that bash completion scripts try to parse bash in bash, which is an even worse idea than trying to parse JSON in bash) I'd argue that Awk is ALSO too weak to parse JSON The following code assumes th…

I don't really buy that shell / awk is "too weak" to deal with JSON, the ecosystem of tools is just fairly immature as most of the shells common tools predate JSON by at least a decade. `jq` being a pretty reasonable addition to the standard set of tools included in environments by default. IMO the real problem is that JSON doesn't work very well at as a because it's core abstraction is objects . It's a pain to deal…

I'd say that awk really is too weak. Awk has a grand total of 2 data types: strings, and associative arrays mapping strings to strings. There is no support for arbitrarily nested data structures. You can simulate them with arrays if you really want to, or you could shell out to jq, but it's definitely swimming upstream.

Most languages aren't quite that bad. Even if they can't handle JSON very ergonomically, almost every language has at least some concept of nesting objects inside other objects.

What about shell? Just like awk, bash and zsh have a limited number of data types (the same two as awk plus non-associative arrays). So arguably it has the same problem. On the other hand, as you say, in shell it's perfectly idiomatic to use external tools, and jq is one such tool, available on an increasing number of systems. So you may as well store JSON data in your string variables and use jq to access it as needed. Probably won't be any slower than the calls to sed or awk or cut that fill out most shell scripts.

Now, personally, I've gotten into the habit of writing shell scripts with minimal use of external tools. If you stick to shell builtins, your script will run much faster. And both bash and zsh have a pretty decent suite of string manipulation tools, including some regex support, so you often don't actually need sed or awk or cut. However, this also rules out jq, and neither shell has any remotely comparable builtin.

But you might reasonably object that if I care about speed, I would be better off using a real programming language!

Re: Parsing JSON in Forty Lines of Awk

#35

Post by HappMacDonald »

One day I wanted to use a TAP parser for the Test Anything Protocol.

But I didn't want to be bogged down by dependencies.. so I didn't want to go nowhere near Python (and pyenv.. and anaconda.. and then probably having to dockerize that for some reason too..) nor nodeJS nor any of that.

Found a bash shell script to parse TAP written by ESR of all people. That sounds fine, I thought. Most everywhere has bash, and there are no other dependencies.

But it was slow. I mean.. painfully, ridiculously slow. Parsing like 400 lines of TAP took almost a minute.

That's when I did some digging and learned about what awk really is. A scripting language that's baked into the POSIX suite. I had heard vaguely about it beforehand, but never realized it had more power than it's sed/ed/cut/etc brethren in the suite.

So I coded up a TAP parser in awk and it went swimmingly, matched the ESR parser's feature set, and ran millions of lines in less than a second. Score! :D

Re: Parsing JSON in Forty Lines of Awk

#36

Post by twoodfin »

Contemplating this, it’s too bad the Unix scripting ecosystem never evolved a tripartite symbiosis of ‘file‘, ‘lex‘, and ‘yacc‘, or similar tools.

That is, one tool to magically identify a file type, one to tokenize it based on that identification, one to correspondingly parse it. All in a streaming/pipe-friendly mode.

Would fit right in, other than the Unix prejudice (nonsensical from Day 0) for LF-separated text records as the “one true format”.

Re: Parsing JSON in Forty Lines of Awk

#37

Post by kristopolous »

twoodfin wrote: post #36

Contemplating this, it’s too bad the Unix scripting ecosystem never evolved a tripartite symbiosis of ‘file‘, ‘lex‘, and ‘yacc‘, or similar tools. That is, one tool to magically identify a file type, one to tokenize it based on that identification, one to correspondingly parse it. All in a streaming/pipe-friendly mode. Would fit right in, other than the Unix prejudice (nonsensical from Day 0) for LF-separated text re…

I guess ... I don't think it really gets you much unless it's 1-pass streaming otherwise we're dealing with entire input buffers and then we're just back to files.

You could argue using the vertical separator | is more syntactically graceful but then it's just a shell argument. There's quite a few radically different shells out there these days like xonsh, murex, and nushell so if simply arranging logic on the screen in a different syntax is what you're looking for then that's probably the way.

Re: Parsing JSON in Forty Lines of Awk

#39

Post by chubot »

alganet wrote: post #12
chubot wrote: post #8

JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character Yes, shell is definitely too weak to parse JSON! (One reason I started https://oils.pub is because I saw that bash completion scripts try to parse bash in bash, which is an even worse idea than trying to parse JSON in bash) I'd argue that Awk is ALSO too weak to parse JSON The following code assumes th…

> Yes, shell is definitely too weak to parse JSON! Parsing is a trivial, rejecting invalid input is trivial, the problem is representing the parsed content in a meaningful way. > bash completion scripts try to parse bash in bash You're talking about ble.sh, right? I investigated it as well. I think they made some choices that eventually led to the parser being too complex, largely due to the problem of representing w…

I was referring to the bash-completion project, the default on Debian/Ubuntu - https://github.com/scop/bash-completion/

But yes, ble.sh also has a shell parser in shell, although it uses a state machine style that's more principled than bash regex / sed crap.

---

Also, distro build systems like Alpine Linux and others tend to parse shell in shell (or with sed).

They often need package metadata without executing package builds, so they do that by trying to parse shell.

In YSH, you will be able to do that with reflection, basically like Lisp/Python/Ruby, rather than ad hoc parsing.

---

I'm glad to hear you can see the effect of the optimizations ! That took a long time :-)

Some more benchmarks here, which I'll write about: https://oils.pub/release/0.33.0/benchmarks.wwz/osh-runtime/

Re: Parsing JSON in Forty Lines of Awk

#40

Post by Brian_K_White »

chubot wrote: post #8

JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character Yes, shell is definitely too weak to parse JSON! (One reason I started https://oils.pub is because I saw that bash completion scripts try to parse bash in bash, which is an even worse idea than trying to parse JSON in bash) I'd argue that Awk is ALSO too weak to parse JSON The following code assumes th…

The same author already had made the more thorough jawk. They explicitly said they wanted a cut down version. It's not illegal to want a cut down version of something.
Post reply on HN