Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting Objects Recursively with Jq

til.simonwillison.net

Post reply on HN
Print view View original

11–20 of 76 posts

Re: Extracting Objects Recursively with Jq

#11

Post by pcr910303 »

A bit offtopic, but I don't see much people knowing/using the Algolia API[0]. It's much better to use than the HN official API[1], since it returns the whole tree data in one request.

Unfortunately (I guess this is a big reason why people don't use it), it doesn't sort the comments – if you need the orders, you'll have to parse HN HTML (or just use the official API).

Still just two requests (the HN site, the Algolia API) is much better than recursively requesting a hundred requests, so I use this approach in my client[2].

[0]: https://hn.algolia.com/api

[1]: https://github.com/HackerNews/API

[2]: https://github.com/goranmoomin/HackerNews

Re: Extracting Objects Recursively with Jq

#12

Post by xcambar »

I love jq for many reasons:

* it is very potent

* it has extensive documentation, written more like parabolas and new-age sorcery than actually helpful content

* it improves your search skills greatly, for many internet results try to give sense to its format

* it gives great satisfaction after you've died-and-retried 300x a whole afternoon for a command you'll only need once.

* it looks cool to use jq instead of [any language you're already familiar with].

Yes, I'm being sarcastic, yet honest. Note that I still use it and recommend it for simpler use cases though.

_[shrug]_

Re: Extracting Objects Recursively with Jq

#13

Post by damagednoob »

xcambar wrote: post #12

I love jq for many reasons: * it is very potent * it has extensive documentation, written more like parabolas and new-age sorcery than actually helpful content * it improves your search skills greatly, for many internet results try to give sense to its format * it gives great satisfaction after you've died-and-retried 300x a whole afternoon for a command you'll only need once. * it looks cool to use jq instead of [an…

I too find the jq syntax arcane and I've found https://jqplay.org/ to be an invaluable help.

Re: Extracting Objects Recursively with Jq

#14

Post by ocimbote »

jcims wrote: post #2

jq seems incredibly powerful. I only find myself using it a few times a year though, and have never been able to conceptualize the syntax enough to use it without prodigious googling.

Yes, jq is very powerful yet incredibly inefficient to work with when you don't master its arcanes.

Re: Extracting Objects Recursively with Jq

#15

Post by hoshsadiq »

I tend to use jq a lot. As others have said, sometimes jq can be hard to grasp. Often it requires multiple attempts to get the correct answer. To make it a little easier for me, I've written a helper function[0] that combines it with fzf[1] to run jq as a REPL on any json. It allows to incrementally alter your DSL without having to continually call jq. This is similar to jid/jiq but a little more powerful. It includes functions to change the preview to output raw, compact (or not), and some other things.

It is essentially similar to jqplay but local.

I didn't use jid/jiq because jid uses go-simplejson, which is nowhere near as powerful as jq, and jiq seemed very buggy when I used it and it felt like it was hacked together. Plus there was no where to change jq's arguments while running it.

I'm sure this function can be improved on, but this has been good enough for me so far.

Also, I run gojq[2] instead of jq. It is a drop-in replacement for jq but is written in Go, and has some improvements over jq such as bug fixes, support for yaml input, and it also provides more helpful error messages.

[0] https://github.com/hoshsadiq/dot_files/blob/master/zshrc.d/m...

[1] https://github.com/junegunn/fzf

[2] https://github.com/itchyny/gojq/

Re: Extracting Objects Recursively with Jq

#16

Post by minxomat »

jcims wrote: post #2

jq seems incredibly powerful. I only find myself using it a few times a year though, and have never been able to conceptualize the syntax enough to use it without prodigious googling.

Think of it just like bash. jq is the ultimate functional language and/or ETL tool. If I look back at larger jq transforms I've written a while ago (e.g. https://git.io/JBSfB) they still make perfect sense to me.

Re: Extracting Objects Recursively with Jq

#17

Post by rattray »

hoshsadiq wrote: post #15

I tend to use jq a lot. As others have said, sometimes jq can be hard to grasp. Often it requires multiple attempts to get the correct answer. To make it a little easier for me, I've written a helper function[0] that combines it with fzf[1] to run jq as a REPL on any json. It allows to incrementally alter your DSL without having to continually call jq. This is similar to jid/jiq but a little more powerful. It include…

This looks cool - would you be willing to share it in a brew-installable format?

Re: Extracting Objects Recursively with Jq

#19

Post by matthewtovbin »

psanford wrote: post #4

JQ is often frustrating when you want to do something non-trivial but you can't figure out how and the documentation is of little help. I think JQ could really benefit from having a classic programming language style "book", like "The AWK Programming Language". JQ is fundamentally a functional programming language with semantics that are not obvious reading its current docs.

Try JSONata instead - http://docs.jsonata.org/overview.html
Post reply on HN