Zq: An easier and faster alternative to jq

From a computer science point of view, what kind of transformations are impossible to express in jmespath but are possible in jq?

I completely agree when your goal is GSD just use the tools you have.

When you have time to sharpen the saw come back and dig into the details of how jq and tools like it work and where their limits are. Looking at the jq builtins[1] can be very enlightening

If you get to the point where your goal is to increase your jq skills I'd recommend looking at the jq questions on Stack Overflow and posting your own solution. Contributing a solution to https://rosettacode.org/wiki/Category:Jq is also good.

1- https://github.com/stedolan/jq/blob/master/src/builtin.jq

Wow, thanks.

Coincidentally, after hearing of a friend's woes dealing with massive amounts of CSV coming from a BPF-instrumental kernel, I played around a bit with integrating Zed and BPF. Just an experimental toy (and the repo is already out of date)...

https://github.com/brimdata/zbpf

The nice thing about Zed here is any value can be a group-by key so it's easy, for example, to use kernel stacks (an array of strings) in a grouping aggregate.

(p.s. for the record, the only thing I have to do with the modern linux BPF system is the tiny vestige of origin story it shares with the original work I did in the BSD kernel around 1990)

So, admitted jq fanboy here, but I found a lot of the criticism from the articale really sensible.

I think jq has a pretty elegant data model, but the syntax is often very clunky to work with.

So here is a half thought-out idea how you might improve the syntax for the "stateful operations" usecase the OP outlined:

I think it's not quite true that different elements of a sequence can never interact. The OP mentioned reduce/foreach, but it's also what any function that takes argument does:

If you have an expression 'foo | bar', then bar is called once for every element foo emits. However, foo could also a function that takes arguments. Then you can specify bar as an argument of foo like this: 'foo(bar)'. In this situation, execution of bar is completely controlled by foo. In particular, foo gets to see all elements that foo emits, not just one each. I believe this is how e.g. [x] can collect all elements of x into an array.

In the same way, you could write a function 'add_all(x)' which calls x and adds up all emitted elements to a sum.

However, this wouldn't help you with collecting all input lines, as there is nothing for you function to "wrap around". Or at least, there used to be nothing, but I think in one of the recent build, an "inputs" function was added, which emits all remaining inputs. So now, you can write e.g. '[., inputs]' to reimplement slurp. In the same way, you could sum up all input lines by writing 'add_all(., inputs)'.

However, this is still ugly and unintuitive to write, so I think introducting some syntactic sugar for this would be useful. E.g., you could imagine a "collect operator", e.g. '>>' which treats everything left of it as the first argument to the function to the right of it.

e.g., writing 'a >> b' would desugar to 'b(a)'.

Writing 'a | b >> c' would desugar to 'c(a | b)'.

Any steps further to the right are not affected:

'a | b >> c | d' would desugar to 'c(a | b) | d'.

Scope to the left could be controlled with parantheses:

'a | (b >> c)' would desugar to 'a | c(b)'.

To make this more useful for aggregating on input lines, you could add a special rule that, if the operator is used with no parantheses, it will implicitly prepend '(., inputs)' as the first step.

So if the entire top-level expression is 'a | b >> c', it would desugar to 'c((., inputs) | a | b)'.

This would make many usecases that require keeping state much more straight-forward. E.g. collecting all the "baz" fields into an array could be written as '.baz >> []' which would desugar to '[(., inputs) | .baz]'

Summing up all the bazzes could be written as '.baz >> add_all' which would desugar to 'add_all((., inputs) | .baz)'

...and so on.

On the other hand, this could also lead to new confusion, as you could also write stuff like '... | (.baz >> map) | ...' which would really mean 'map(.baz)' or 'foo >> bar >> baz' which would desugar to the extremely cryptic expression 'baz((., inputs) | bar((., inputs) | foo))'. So I'm not quite sure.

Any thoughts about the idea?

I dunno how to speak to your "computer science" part, but pragmatically anything that requires a "backreference", because unlike with JSONPath (and, of course, jq) there are no "root object" references

    $ printf '{"a": {"b":"c", "d":["d0","d1"]}}' | jq -r '[ .a as $a | $a.d[] | {x: ., y: $a.b}]'
    [
      {
        "x": "d0",
        "y": "c"
      },
      {
        "x": "d1",
        "y": "c"
      }
    ]

and I realize this isn't as pure CS-y as you were asking, but this syntax is hell on quoting

    $ printf '["a","b"]' | jp -u 'join(`"\n"`, @)'
    # vs
    $ printf '["a","b"]' | jq -r 'join("\n")'