just use nushell man
Parsing JSON in Forty Lines of Awk
41–50 of 62 posts
Re: Parsing JSON in Forty Lines of Awk
#42[deleted]
Re: Parsing JSON in Forty Lines of Awk
#43One day I wanted to use a TAP parser for the Test Anything Protocol. But I didn't want to be bogged down by dependencies.. so I didn't want to go nowhere near Python (and pyenv.. and anaconda.. and then probably having to dockerize that for some reason too..) nor nodeJS nor any of that. Found a bash shell script to parse TAP written by ESR of all people. That sounds fine, I thought. Most everywhere has bash, and ther…
For the record, "python-without-extra-dependencies" is a thing and a very nice one too. I always prefer it over awk.
Highly recommend to everyone - plenty of "batteries" included, like json parser, basic http client and even XML parser, and no venv/conda required. Very good forward compatibility. Fast (compared to bash).
Re: Parsing JSON in Forty Lines of Awk
#44I use flex. Faster than awk or python. It produces relatively seems to be available "everywhere" because it is a build requirement for so many software programs. For exampole, NetBSD toolchain includes it. It is a build requirement for Linux kernel.^1 I have even used it with Interix SFU on Windows before WSL existed.
I do not use jq. Too complicated for me. Overkill. I created statically-linked program less than half the size of official statically-linked jq that is adequate for own needs. flex is a build requirement for jq.
1. https://www.kernel.org/doc/Documentation/admin-guide/quickly...
Re: Parsing JSON in Forty Lines of Awk
#45Earlier quoted context omitted.
> Yes, shell is definitely too weak to parse JSON! Parsing is a trivial, rejecting invalid input is trivial, the problem is representing the parsed content in a meaningful way. > bash completion scripts try to parse bash in bash You're talking about ble.sh, right? I investigated it as well. I think they made some choices that eventually led to the parser being too complex, largely due to the problem of representing w…
I was referring to the bash-completion project, the default on Debian/Ubuntu - https://github.com/scop/bash-completion/ But yes, ble.sh also has a shell parser in shell, although it uses a state machine style that's more principled than bash regex / sed crap. --- Also, distro build systems like Alpine Linux and others tend to parse shell in shell (or with sed). They often need package metadata without executing packa…
You may well already be aware, but just in case you aren't, your bin-true benchmark mostly measures dynamic loader overhead, not fork-exec (e.g., I got 5.2X faster using a musl-gcc statically linked true vs. glibc dynamic coreutils). { Kind of a distro/cultural thing what you want to measure (static linking is common on Alpine Linux, BSDs, less so on most Linux), but good to know about the effect. }
Re: Parsing JSON in Forty Lines of Awk
#46Re: Parsing JSON in Forty Lines of Awk
#47Earlier quoted context omitted.
Or sticking with awk, I have this bash alias to remove excess whitespace that is just: awk '{$1=$1};1'
what does the 1 at the end do? make awk print all lines? I'm a bit rusty with my awk.
It’s a condition. 1 is true-ish, so it’s a condition that is always true. The default action in awk is print, so it’s the same as:
'{$1=$1}1{print}'
'{$1=$1}{print}'
Re: Parsing JSON in Forty Lines of Awk
#48One day I wanted to use a TAP parser for the Test Anything Protocol. But I didn't want to be bogged down by dependencies.. so I didn't want to go nowhere near Python (and pyenv.. and anaconda.. and then probably having to dockerize that for some reason too..) nor nodeJS nor any of that. Found a bash shell script to parse TAP written by ESR of all people. That sounds fine, I thought. Most everywhere has bash, and ther…
For the record, "python-without-extra-dependencies" is a thing and a very nice one too. I always prefer it over awk. Highly recommend to everyone - plenty of "batteries" included, like json parser, basic http client and even XML parser, and no venv/conda required. Very good forward compatibility. Fast (compared to bash).
Can you explain more or provide more information/links?
Re: Parsing JSON in Forty Lines of Awk
#49Earlier quoted context omitted.
I was referring to the bash-completion project, the default on Debian/Ubuntu - https://github.com/scop/bash-completion/ But yes, ble.sh also has a shell parser in shell, although it uses a state machine style that's more principled than bash regex / sed crap. --- Also, distro build systems like Alpine Linux and others tend to parse shell in shell (or with sed). They often need package metadata without executing packa…
You may well already be aware, but just in case you aren't, your bin-true benchmark mostly measures dynamic loader overhead, not fork-exec (e.g., I got 5.2X faster using a musl-gcc statically linked true vs. glibc dynamic coreutils). { Kind of a distro/cultural thing what you want to measure (static linking is common on Alpine Linux, BSDs, less so on most Linux), but good to know about the effect. }
Yup, I added an osh-static column there, because I know dynamic linking slows things down. (With the latest release, we have a documented build script to make osh-static, which I tested with GNU libc and musl: https://oils.pub/release/latest/doc/help-mirror.html)
Although I think the CALLING process (the shell) being dynamically linked affects the speed too, not just the CALLED process (/bin/true)
I'd like to read an analysis of why that is! And some deeper measurements
Re: Parsing JSON in Forty Lines of Awk
#50Earlier quoted context omitted.
> Yes, shell is definitely too weak to parse JSON! Parsing is a trivial, rejecting invalid input is trivial, the problem is representing the parsed content in a meaningful way. > bash completion scripts try to parse bash in bash You're talking about ble.sh, right? I investigated it as well. I think they made some choices that eventually led to the parser being too complex, largely due to the problem of representing w…
I was referring to the bash-completion project, the default on Debian/Ubuntu - https://github.com/scop/bash-completion/ But yes, ble.sh also has a shell parser in shell, although it uses a state machine style that's more principled than bash regex / sed crap. --- Also, distro build systems like Alpine Linux and others tend to parse shell in shell (or with sed). They often need package metadata without executing packa…
> uses a state machine style
That's the way to go. I don't even consider other shallow and ad-hoc approaches as actually parsing it.
I've been working on a state-machine based parser of my own. It's hard, I'm targetting very barebones interpreters such as posh and dash. Here's what it looks like
https://gist.github.com/alganet/23df53c567b8a0bf959ecbc7b689...
(not fully working example, but it gives an idea of what pure POSIX shell parsing looks like, ignore the aliases, they'll not be in the final version).
> I'm glad to hear you can see the effect of the optimizations ! That took a long time :-)
Yep, been testing osh since 0.9! Still a long way to go to catch up with ksh93 though, it's the fastest of all shells (even dash) by a wide margin.
By beating bash, you also have beaten zsh (it's one of the slowest shells around).