Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zq: An easier and faster alternative to jq

brimdata.io

Post reply on HN
Print view View original

71–80 of 237 posts

Re: Zq: An easier and faster alternative to jq

#71

Post by AcerbicZero »

I'm pretty new to jq (maybe 2 years of exposure) but from my perspective - on some level, jq does to json what powershell does to everything windows, except powershell gives me the get-member cmdlet, so when I don't know what is even in my object, I can explore.

Sometimes jq -r '.[]' works, but its all just trial and error. I use plenty of jq in my scripts, but I can never seem to visualize how jq looks at the data. I just have to toss variations of '.[whateveriwant].whatever[.want.]' until something works....I suppose the root of my complaint is that jq does not do a good job of teaching you to use jq. It either works, or gives you nothing, and while I've learned to work around that, I'll try anything that claims to be even 1% better than jq.

Re: Zq: An easier and faster alternative to jq

#72

Post by bblb »

brushfoot wrote: post #5

The name of its corporate progenitor may leave a bad taste in some mouths, but I highly recommend PowerShell for this sort of thing. It's cross platform, MIT licensed, and comes with excellent JSON parsing and querying capabilities. Reading, parsing, and querying JSON to return all red cars: Get-Content cars.json | ConvertFrom-Json | ? { $_.color -eq 'red' } The beauty of this is that the query syntax applies not jus…

PowerShell is "Python interactive done right". It's too bad it has a bad rap in open source community and it might never get the traction it really deserves. Sure it has it's downsides, which tech doesn't, but PowerShell has solved so many issues and annoyances with the shells that we've been used to, that it still comes out as the winner.

I've been using it since day one from 2006, every single day. It has come a long way and the current PS7 is the best shell experience there is. Hands down no contest.

Snover's passionate early presentation about the PS pipeline is a pretty cool tech video. https://www.youtube.com/watch?v=325kY2Umgw8

Re: Zq: An easier and faster alternative to jq

#73

Post by ElectricalUnion »

mschuster91 wrote: post #48
ElectricalUnion wrote: post #40

Earlier quoted context omitted.

> What I would actually expect of Microsoft is to follow the Apple way: have one single central place, ideally at setup and later in the System Preferences, where tracking, analytics and other optional crap can be disabled system-wide. This is still GDPR non-compliant, you should have a central place to _opt-in_ tracking, analytics and other optional crap if you so desire.

So what? You can opt-in to tracking in the macOS System Preferences, pane "security and data protection", tab "Privacy" at any time you wish should you not have done so during the macOS onboarding process. In Debian, you can opt-in at setup time or any later time with a simple "dpkg-reconfigure popularity-contest" (even though that one isn't fully GDPR-compliant as you can't easily read what exactly is being done fro…

> So what? You can opt-in to tracking in the macOS System Preferences, pane "security and data protection", tab "Privacy" at any time you wish should you not have done so during the macOS onboarding process.

You cannot opt-in. You can go to `System Preferences > Security & Privacy > Analytics & Improvements` and opt-out, but the default is not opt-in.

Re: Zq: An easier and faster alternative to jq

#74

Post by j1elo »

psacawa wrote: post #6

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lackin…

Sadly very few authors seem to acknowledge or even know that github wiki pages are not indexed by search engines so if it wasn't for third-party sites like github-wiki-see.page (which could stop working at any time) their contents would be undiscoverable by the very same people they are usually intended...

Re: Zq: An easier and faster alternative to jq

#75

Post by hiram112 »

Beltalowda wrote: post #57
psacawa wrote: post #6

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lackin…

> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forg…

Bingo.

There are at least a dozen tools and languages and syntaxes that I've used sporadically over the years - awk, sed, bash, Mongo, perl, etc. I don't use them often enough to remember exactly how they work, and so I always have to spend a few hours reviewing manuals or old code repos or an O'Reilly book.

But if I do end up using it for a few days in a row, it starts to make sense, and I improve each time I use it.

But not with jq.

It just does not make sense to my brain, no matter how many times I've had to use it. Every single time I need to use it, it requires finding some Stack Exchange or blog and just copying and pasting. Even after seeing the solution, rarely do I then really understand why or how it works. Nor can I often take that knowledge and apply it to similar problems.

About the only other syntax or language that gives me such problems is Elastic Search DSL.

Re: Zq: An easier and faster alternative to jq

#76

Post by spudlyo »

kitd wrote: post #39
weinzierl wrote: post #12

jq is incredibly powerful and I'm using it more and more. Even better, there is a whole ecosystem of tools that are similar or work in conjunction with jq: * jq (a great JSON-wrangling tool) * jc (convert various tools’ output into JSON) * jo (create JSON objects) * yq (like jq, but for YAML) * fq (like jq, but for binary) * htmlq (like jq, but for HTML) List shamelessly stolen from Julia Evans[1]. For live links see…

Also highly recommended is gron [0], to make json easily searchable [0] https://github.com/TomNomNom/gron

Most of the time I can get what I need with gron and traditional UNIX tools, without needing to reach for jq, and without having to re-learn its somewhat arcane syntax.

Re: Zq: An easier and faster alternative to jq

#77

Post by ts0000 »

Beltalowda wrote: post #57
psacawa wrote: post #6

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lackin…

> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forg…

Interesting, for me it's the exact opposite.

I've tried a couple of times to get into awk, but still find the syntax arcane.

Re: Zq: An easier and faster alternative to jq

#78

Post by adamgordonbell »

psacawa wrote: post #6

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lackin…

I found it hard to approach at first, but I think it was just the lack of material that worked through simple examples step by step.

I ended up writing my own guide to it, that in my unbiased opinion makes it easier to get the point where in-depth examples and language descriptions are easier to understand.

Edit: Oh, wow, it's even mentioned in this article. Maybe I should read before commenting.

https://earthly.dev/blog/jq-select/

Re: Zq: An easier and faster alternative to jq

#79

Post by micimize »

Their syntax comparison under "So you like chocolate or vanilla?" is disingenuous. You can do variable assignment and array expansion in jq:

  expand_vals_into_independent_records='
    .name as $name | .vals[] | { name: $name, val: . }
  '
  echo '{"name":"foo","vals":[1,2,3]} {"name":"bar","vals":[4,5]}' |
    jq "$expand_vals_into_independent_records"
Also, generally, not a fan of the tone of this article.

Re: Zq: An easier and faster alternative to jq

#80

Post by justinsaccount »

cosmiccatnap wrote: post #14

I would love to see what jq looks like on something like a 1mil line Json vs this. In my experience jq syntax is fine and I've not ran into a performance issue on any one file but I seem to see a jq clone every few months on here so someone seems to need that, or maybe it's just the new volume slider problem who knows.

jq performance is pretty terrible. Here I'm going to do something super simple like pull out a single field out of a large log file:

  $ wc -l big.log 
    979400 big.log

  $ du -hs big.log 
  570M big.log
`count` is a small program that counts lines on stdin. like `sort|uniq -c |sort -n`

jq takes 12 seconds:

  $ time cat big.log |jq -cr .method |~/bin/count 
  848000 GET
  94800 POST
  34000 HEAD
  2400 OPTIONS
  200 null

  real 0m12.381s
  user 0m12.427s
  sys 0m0.333s
my tool takes .5 seconds

  $ time cat big.log |~/bin/jj method |~/bin/count 
  848000 GET
  94800 POST
  34000 HEAD
  2400 OPTIONS
  200 

  real 0m0.466s
  user 0m0.512s
  sys 0m0.198s
`jj` is a little tool I wrote that uses https://github.com/buger/jsonparser
Post reply on HN