Live data from Hacker News

Extracting Objects Recursively with Jq

til.simonwillison.net

31–40 of 76 posts

Re: Extracting Objects Recursively with Jq

#31

I tend to use jq a lot. As others have said, sometimes jq can be hard to grasp. Often it requires multiple attempts to get the correct answer. To make it a little easier for me, I've written a helper function[0] that combines it with fzf[1] to run jq as a REPL on any json. It allows to incrementally alter your DSL without having to continually call jq. This is similar to jid/jiq but a little more powerful. It include…

That’s cool! What’s the use of FZF? Isn’t it a fuzzy finder, what are you searching for in a jq REPL?

It only uses the FZF's preview. The suggestions is completely empty. I tried to find an alternative as FZF has no way of disabling the selector window, but I was unable to find anything that was good enough for this.

I considered forking jid/jiq and using gojq as a library, but I ended up not going down that route because of reasons that I cannot remember. I also considered using a tui or something but FZF has so much already implemented and has a lot of it right, and I didn't particularly feel like re-inventing the wheel.

Re: Extracting Objects Recursively with Jq

#32
post #17

I tend to use jq a lot. As others have said, sometimes jq can be hard to grasp. Often it requires multiple attempts to get the correct answer. To make it a little easier for me, I've written a helper function[0] that combines it with fzf[1] to run jq as a REPL on any json. It allows to incrementally alter your DSL without having to continually call jq. This is similar to jid/jiq but a little more powerful. It include…

This looks cool - would you be willing to share it in a brew-installable format?

Not particularly. I don't use a Mac, but I'd be happy to separate out the function into it's own script so it can be downloaded and put in your $PATH. I personally use zinit to manage individual files from random repos.

Re: Extracting Objects Recursively with Jq

#33
To better appreciate the structure of the document the author is dealing with (and to cast a bit of light on which words are variables in the document and which are `jq` syntax. I offer a shameless plug to a one liner

well, I would but the result is "too long for a HN comment" so here a bunch is sniped out of the middle (unedited the result would currently be 140 lines)

  curl -s  https://hn.algolia.com/api/v1/items/27941108 | ~/bin/json2jqpath.jq  

  .
  .author
  .children
  .children|.[]
  .children|.[]|.author
  .children|.[]|.children
  .children|.[]|.children|.[]
  .children|.[]|.children|.[]|.author
  .children|.[]|.children|.[]|.children
  .children|.[]|.children|.[]|.children|.[]
  .children|.[]|.children|.[]|.children|.[]|.author
  .children|.[]|.children|.[]|.children|.[]|.children
  .children|.[]|.children|.[]|.children|.[]|.children|.[]
    
  ...  
  
  .children|.[]|.children|.[]|.created_at
  .children|.[]|.children|.[]|.created_at_i
  .children|.[]|.children|.[]|.id
  .children|.[]|.children|.[]|.options
  .children|.[]|.children|.[]|.parent_id
  .children|.[]|.children|.[]|.points
  .children|.[]|.children|.[]|.story_id
  .children|.[]|.children|.[]|.text
  .children|.[]|.children|.[]|.title
  .children|.[]|.children|.[]|.type
  .children|.[]|.children|.[]|.url
  .children|.[]|.created_at
  .children|.[]|.created_at_i
  .children|.[]|.id
  .children|.[]|.options
  .children|.[]|.parent_id
  .children|.[]|.points
  .children|.[]|.story_id
  .children|.[]|.text
  .children|.[]|.title
  .children|.[]|.type
  .children|.[]|.url
  .created_at
  .created_at_i
  .id
  .options
  .parent_id
  .points
  .story_id
  .text
  .title 
  .type
  .url

[0] https://github.com/TomConlin/json_to_paths

Re: Extracting Objects Recursively with Jq

#37
post #33

To better appreciate the structure of the document the author is dealing with (and to cast a bit of light on which words are variables in the document and which are `jq` syntax. I offer a shameless plug to a one liner well, I would but the result is "too long for a HN comment" so here a bunch is sniped out of the middle (unedited the result would currently be 140 lines) curl -s https://hn.algolia.com/api/v1/items/279…

Thanks for linking this! I've wanted exactly this script, many times.

Re: Extracting Objects Recursively with Jq

#38
JSON data is also a valid Prolog term, and the declarative programming language Prolog is ideally suited for handling tree-shaped data.

Using for example Scryer Prolog, we can conveniently relate the data to a flat list of items with Prolog's built-in grammar mechanism, definite clause grammars (DCGs):

    flat_json(JSON) -->
            { JSON = {A,B,C,D,E,F,_:Cs} },
            [{A,B,C,D,E,F}],
            flat_items(Cs).

    flat_items([]) --> [].
    flat_items([I|Is]) -->
            { I = {(A,B,C,D,E,_:Cs)} },
            [{A,B,C,D,E}],
            flat_items(Cs),
            flat_items(Is).
Sample query, using the example JSON data from the article:

     ?- JSON = {
        "id": 27941108,
        "created_at": "2021-07-24T14:15:05.000Z",
        "type": "story",
        "author": "edward",
        "title": "Fun with Unix domain sockets",
        "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/",
        "children": [
            {
                "id": 27942287,
                "created_at": "2021-07-24T16:31:18.000Z",
                "type": "comment",
                "author": "DesiLurker",
                "text": "

one lesser known...", "children": [] }, { "id": 27944615, "created_at": "2021-07-24T21:26:33.000Z", "type": "comment", "author": "galaxyLogic", "text": "

I read this from Wikipedia...", "children": [ { "id": 27944746, "created_at": "2021-07-24T21:49:07.000Z", "type": "comment", "author": "hughrr", "text": "

Yes although I ...", "children": [] } ] } ] }, phrase(flat_json(JSON), Cs), maplist(portray_clause, Cs).

yielding the flat list of entries, as desired:

    [{("id":27941108,"created_at":"2021-07-24T14:15:05.000Z","type":"story","author":"edward",...)},
     {("id":27942287,"created_at":"2021-07-24T16:31:18.000Z","type":"comment",...)},
     {("id":27944615,"created_at":"2021-07-24T21:26:33.000Z","type":"comment",...)},
     {("id":27944746,"created_at":"2021-07-24T21:49:07.000Z","type":"comment",...)}]

Re: Extracting Objects Recursively with Jq

#39
post #38

JSON data is also a valid Prolog term, and the declarative programming language Prolog is ideally suited for handling tree-shaped data. Using for example Scryer Prolog, we can conveniently relate the data to a flat list of items with Prolog's built-in grammar mechanism, definite clause grammars (DCGs): flat_json(JSON) --> { JSON = {A,B,C,D,E,F,_:Cs} }, [{A,B,C,D,E,F}], flat_items(Cs). flat_items([]) --> []. flat_item…

Anyway to avoid the need to enumerate the json fields that come before children with those placeholders? Otherwise, it will be brittle to modifications.

Prolog was the one language I couldn’t get my head around in the programming languages class I took at school.

Re: Extracting Objects Recursively with Jq

#40
post #4

JQ is often frustrating when you want to do something non-trivial but you can't figure out how and the documentation is of little help. I think JQ could really benefit from having a classic programming language style "book", like "The AWK Programming Language". JQ is fundamentally a functional programming language with semantics that are not obvious reading its current docs.

Try JSONata instead - http://docs.jsonata.org/overview.html

It doesn't seem to provide a command-line tool, which is the point of jq
Post reply on HN