Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zq: An easier and faster alternative to jq

brimdata.io

Post reply on HN
Print view View original

51–60 of 237 posts

Re: Zq: An easier and faster alternative to jq

#51

Post by johnday »

diehunde wrote: post #22

Pardon my ignorance, but would I spend time learning something like jq or zq when it only takes me a couple of minutes to develop a script using some high-level language? I've had to process complex JSON files in the past, and a simple Python script gets the job done, and the syntax is much more familiar and easier to memorize. Is there a use case I'm missing?

Suppose you write a shell script which is intended for use among colleagues as part of a pipeline.

In many cases, the most appropriate and useful tool for the job would be jq - one line in the shell script corresponding to the required data transform, calling out to `jq`, which already has a reasonable user base and documentation, and could be trivially replaced by anyone if the business needs change.

Re: Zq: An easier and faster alternative to jq

#53

Post by eru »

gcmeplz wrote: post #50

I like using `jq` to create line-delimited JSON and then using a language I know well (Node) to process it after that point. I find `jq '.[] | select(.location=="Stockholm")'` less readable than something like `nq --filter '({location}) => location === "Stockholm"'` because I'm much more used to Node syntax. - https://github.com/thisredone/rb is a widely used ruby version of this idea - https://github.com/KelWill/nq#…

By Node, you mean JavaScript?

If yes, it's fascinating to me, that jq is so powerful, it's even useful when handling JavaScript Object Notation in JavaScript.

Re: Zq: An easier and faster alternative to jq

#54

Post by eru »

meepmorp wrote: post #31
diehunde wrote: post #22

Pardon my ignorance, but would I spend time learning something like jq or zq when it only takes me a couple of minutes to develop a script using some high-level language? I've had to process complex JSON files in the past, and a simple Python script gets the job done, and the syntax is much more familiar and easier to memorize. Is there a use case I'm missing?

The same thing could be said for grep, or really any other utility that can have its functionality reproduced in a programing language.

Indeed! Jq is basically something like grep for JSON.

It might actually make sense to embed jq functionality into your favourite language (as a library or so), as it is quite a nice and well-chosen set of functionality.

Re: Zq: An easier and faster alternative to jq

#55

Post by mattnibs »

p5a0u9l wrote: post #24
algesten wrote: post #3

I don't get it. "Instead of learning jq DSL, learn zq DSL". To me they look similarly complicated and the examples stresses certain aggregation operations that are harder to do in jq (due to it being stateless).

Yes, but fortunately, your efforts will pay dividends when parsing all the 'z*' boutique formats that it supports, zson, zst, zng, the list goes on. /s

Not sure if this came across in the article, but all the "boutique" z* formats are all representations of the same zed model https://zed.brimdata.io/docs/formats/zed/

Re: Zq: An easier and faster alternative to jq

#56

Post by eru »

johnthuss wrote: post #37
diehunde wrote: post #22

Pardon my ignorance, but would I spend time learning something like jq or zq when it only takes me a couple of minutes to develop a script using some high-level language? I've had to process complex JSON files in the past, and a simple Python script gets the job done, and the syntax is much more familiar and easier to memorize. Is there a use case I'm missing?

There is certainly a learning curve with jq that can put people off. The attraction is that the end result is a very small amount of code that does only one thing: parse a JSON file, rather than invoking an external script that might send many HTTP requests or launch a missile. As the complexity of the input JSON grows or the complexity of your processing, it does makes sense to leave jq behind for a higher level lan…

I agree with most of what you say.

I disagree with 'leaving for a higher level language'. Jq is an extremely high level language.

What it is _not_ is a general purpose language.

Re: Zq: An easier and faster alternative to jq

#57

Post by Beltalowda »

psacawa wrote: post #6

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lackin…

> It's not as arcane as it seems.

The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically.

In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember.

Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forgetting it.

Re: Zq: An easier and faster alternative to jq

#58

Post by eru »

folkrav wrote: post #27
diehunde wrote: post #22

Pardon my ignorance, but would I spend time learning something like jq or zq when it only takes me a couple of minutes to develop a script using some high-level language? I've had to process complex JSON files in the past, and a simple Python script gets the job done, and the syntax is much more familiar and easier to memorize. Is there a use case I'm missing?

Honestly, I've only really used `jq` to quickly parse JSON structures in interactive sessions e.g curl -s http://foo.bar | jq .some.nested.value Anything more complicated I would indeed go for writing a proper script.

Don't tell anyone, but jq is secretly a pretty well thought out functional programming language.

Re: Zq: An easier and faster alternative to jq

#59

Post by brushfoot »

mschuster91 wrote: post #33
brushfoot wrote: post #23

Earlier quoted context omitted.

To me a telemetry opt-out is a small price to pay for what PowerShell brings to the table, but to each their own. > There might be other components, now or in the future, that also send data to Microsoft Of course. Do your due diligence on whatever you install. No tool should be exempt from that.

> Do your due diligence on whatever you install. No tool should be exempt from that. That's a ridiculous take. 99% of users don't understand what all that technobabble in a typical EULA means, they will just go for the option they are nudged to (which is why first the courts and now enforcement agencies are stepping up their game against that practice [1]). The way that the GDPR expects stuff to be handled is by gett…

> That's a ridiculous take

Then it befits a ridiculous state of affairs. It would be great to have the standards you suggest, and it's a shame that we don't. But that doesn't change the fact that we don't, and because we don't, we need to do due diligence on the tools we install.

Re: Zq: An easier and faster alternative to jq

#60

Post by chriswarbo »

weinzierl wrote: post #12

jq is incredibly powerful and I'm using it more and more. Even better, there is a whole ecosystem of tools that are similar or work in conjunction with jq: * jq (a great JSON-wrangling tool) * jc (convert various tools’ output into JSON) * jo (create JSON objects) * yq (like jq, but for YAML) * fq (like jq, but for binary) * htmlq (like jq, but for HTML) List shamelessly stolen from Julia Evans[1]. For live links see…

The yq tool also provides 'xq', which works on XML :)
Post reply on HN