Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gojq: Pure Go Implementation of Jq

github.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Gojq: Pure Go Implementation of Jq

#11

Post by cube2222 »

This looks quite cool! I'm not sure though why I would use this over the original jq. However, I can definitely see the value in embedding this into my own applications, to provide jq scripting inside of them.

Shameless plug: As I'm not a fan of the jq syntax, I've created jql[0] as an alternative to it. It's also written in Go and presents a lispy continuation-based query language (it sounds much scarier than it really is!). This way it has much less special syntax than jq and is - at least to me - much easier to compose for common day-to-day JSON manipulation (which is the use case it has been created for; there are definitely many features of jq that aren't covered by it).

It might seem dead, as it hasn't seen any commit in ages, but to me it's just finished, I still use it regularly instead of jq on my local dev machine. Check it out if you're not a fan of the jq syntax.

[0]: https://github.com/cube2222/jql

Re: Gojq: Pure Go Implementation of Jq

#12

Post by silverwind »

simonw wrote: post #2

"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices. I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON a…

Agree, this is deterring me from this tool. Many languages/tools nowadays guarantee object key order which is convenient in many ways.

Re: Gojq: Pure Go Implementation of Jq

#13

Post by tialaramex »

c2h5oh wrote: post #7
simonw wrote: post #6

Earlier quoted context omitted.

I used to have the exact same problem with Python, until Python 3.7 made maintaining sort order a feature of the language: https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/

Go actually went in the other direction for a bunch of reasons (e.g. hash collision dos) and made key order quasi-random when iterating. Small maps used to maintain order, but a change was made to randomize that so people didn't rely on that and get stung when their maps got larger: https://github.com/golang/go/issues/6719

Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster.

It's like if you did such a bad job making a drag racer that the street legal model of the same car was substantially faster over a quarter mile despite also having much better handling and reliability.

In some communities the reaction would have been to write a good unordered dict which would obviously be even faster, but since nobody is exactly looking for the best possible performance from Python, they decided that ordered behaviour was worth the price, and it's not as though existing Python programmers could complain since it was faster than what they'd been tolerating previously.

Randomizing is the other choice if you actually want your maps to be fast and want to resist Hyrum's law, but see the absl experience - they initially didn't bother to randomize tiny maps but then the order of those tiny maps changed for technical reasons and... stuff broke. Because hey, in testing I made six of this tiny map, they always had the same order therefore (ignoring the documentation imploring me not to) I shall assume the order is always the same...

Re: Gojq: Pure Go Implementation of Jq

#14

Post by fwip »

eropple wrote: post #4
fwip wrote: post #3

Not implementing key-sorting is a curious decision: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. I f…

I agree with you that there's value to sorted keys from a presentational standpoint (we are not beep-boop robots, humans have to read this stuff too), but now there also exists a JSON canonicalization RFC that tools can/should follow (with all the usual caveats about canonicalization being fraught): https://www.rfc-editor.org/rfc/rfc8785

Thank you for letting me know! I hadn't thought to look.

Re: Gojq: Pure Go Implementation of Jq

#15

Post by Someone »

renewiltord wrote: post #8
fwip wrote: post #3

Not implementing key-sorting is a curious decision: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. I f…

Could pipe through gron and sort to resort

That helps when you want to sort by key, but not when you want to keep the order of object keys as in the input file.

Re: Gojq: Pure Go Implementation of Jq

#16

Post by alecthomas »

tialaramex wrote: post #13
c2h5oh wrote: post #7

Earlier quoted context omitted.

Go actually went in the other direction for a bunch of reasons (e.g. hash collision dos) and made key order quasi-random when iterating. Small maps used to maintain order, but a change was made to randomize that so people didn't rely on that and get stung when their maps got larger: https://github.com/golang/go/issues/6719

Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. It's like if you did such a bad job making a drag racer that the street legal model of the same car was substantially faster over a quarter mile despite also having much better handling and reliability. In some communities the reaction would have been to write a good unordered d…

> Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster.

I've never heard that before and it would be really surprising, given that Python's builtin dict is used for everything from local symbol to object field lookup. Do you have more information?

Re: Gojq: Pure Go Implementation of Jq

#17

Post by lapser »

silverwind wrote: post #12
simonw wrote: post #2

"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices. I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON a…

Agree, this is deterring me from this tool. Many languages/tools nowadays guarantee object key order which is convenient in many ways.

For what it's worth, JSON Objects are not guaranteed to be ordered. Maps in many different languages are implemented without an order.

Re: Gojq: Pure Go Implementation of Jq

#18

Post by brundolf »

lapser wrote: post #10

I have actually fully replaced my jq installation with gojq (including an `ln -s gojq jq`) for a few years, and no script has broken so far. I'm super impressed by the jq compatibility. If you are going down this route, do be careful with performance. I don't know which is more performant as I've never really had to work with large data sets, but I can't help but feel jq will be faster than gojq in such case. I have…

It's very possible it could be faster; jq seems to actually be fairly unoptimized. This implementation in OCaml was featured on HN a while back and it trashes the original jq in performance: https://github.com/davesnx/query-json

After seeing that one I did my own (less-complete) version in Rust and managed to squeeze out even more performance in the operations it supports: https://github.com/brundonsmith/jqr

Re: Gojq: Pure Go Implementation of Jq

#19

Post by mdaniel »

eropple wrote: post #4
fwip wrote: post #3

Not implementing key-sorting is a curious decision: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. I f…

I agree with you that there's value to sorted keys from a presentational standpoint (we are not beep-boop robots, humans have to read this stuff too), but now there also exists a JSON canonicalization RFC that tools can/should follow (with all the usual caveats about canonicalization being fraught): https://www.rfc-editor.org/rfc/rfc8785

I guess "Informational" is better than /dev/null, but unless everyone adopts it doesn't that run the risk of it just being My Favorite Canonicalization™?

Either way, I'm guessing if the gojq author has that much heartburn about implementing --sort-keys, --canonical is just absolutely off the table :-(

Post reply on HN