Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gojq: Pure Go Implementation of Jq

github.com

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Gojq: Pure Go Implementation of Jq

#71

Post by Someone »

rfiat wrote: post #69
Someone wrote: post #61

Earlier quoted context omitted.

No, it isn’t. “gojq does not keep the order of object keys” isn’t about ordering keys consistently across runs, it’s about keeping them in the order of the input file.

GP is correct, per the README: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. And later in the same fi…

No, they aren’t. haasted replied to simonw’s

> "gojq does not keep the order of object keys" is a bit disappointing

with

> “I bet it's an artifact of Go having a randomized iteration order over maps. Getting a deterministic ordering requires extra work.”

But deterministic iteration order doesn’t imply that the order of keys is kept the same. There are map implementations that keep iteration follow insertion order, but the canonical map does not guarantee that. https://en.wikipedia.org/wiki/Associative_array#Hash_table_i...:

“The most frequently used general purpose implementation of an associative array is with a hash table: an array combined with a hash function that separates each key into a separate "bucket" of the array“

Such implementations iterate over maps in order of hash value (and hash collisions may or may not follow (reverse) insertion order)

Re: Gojq: Pure Go Implementation of Jq

#72

Post by oever »

brundolf wrote: post #18
lapser wrote: post #10

I have actually fully replaced my jq installation with gojq (including an `ln -s gojq jq`) for a few years, and no script has broken so far. I'm super impressed by the jq compatibility. If you are going down this route, do be careful with performance. I don't know which is more performant as I've never really had to work with large data sets, but I can't help but feel jq will be faster than gojq in such case. I have…

It's very possible it could be faster; jq seems to actually be fairly unoptimized. This implementation in OCaml was featured on HN a while back and it trashes the original jq in performance: https://github.com/davesnx/query-json After seeing that one I did my own (less-complete) version in Rust and managed to squeeze out even more performance in the operations it supports: https://github.com/brundonsmith/jqr

Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running on large files like

https://dumps.wikimedia.org/wikidatawiki/entities/latest-all...

A fast tool to split a json file like that into a format with one json file per line would already help a lot.

Re: Gojq: Pure Go Implementation of Jq

#73

Post by brundolf »

oever wrote: post #72
brundolf wrote: post #18

Earlier quoted context omitted.

It's very possible it could be faster; jq seems to actually be fairly unoptimized. This implementation in OCaml was featured on HN a while back and it trashes the original jq in performance: https://github.com/davesnx/query-json After seeing that one I did my own (less-complete) version in Rust and managed to squeeze out even more performance in the operations it supports: https://github.com/brundonsmith/jqr

Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running on large files like https://dumps.wikimedia.org/wikidatawiki/entities/latest-all... A fast tool to split a json file like that into a format with one json file per line would already help a lot.

Mine can :)

Re: Gojq: Pure Go Implementation of Jq

#74

Post by nerdponx »

lemagedurage wrote: post #59
nerdponx wrote: post #45

Earlier quoted context omitted.

The problem with Python here is that CPython is not only the reference implementation but the de-facto specification. So dicts are still "supposed to be" unordered collections, but now dicts must also preserve insertion order as per the docs and the reference implementation, so now all alternative implementations must also conform to this even if it doesn't make sense for them to conform to it, or they must specifica…

Since Python 3.7 preservering insertion-order is part of the language specification. "the insertion-order preservation nature of dict objects has been declared to be an official part of the Python language spec." https://docs.python.org/3/whatsnew/3.7.html

Right, but that's kind of my point. Adding it to the language spec now creates an additional and frankly somewhat unnecessary point of compliance for other implementations. Python is already so damn big and complicated, my opinion is that we shouldn't makes its spec even more complicated, even if its reference implementation adds more features like this.

Re: Gojq: Pure Go Implementation of Jq

#75

Post by rfiat »

Someone wrote: post #71
rfiat wrote: post #69

Earlier quoted context omitted.

GP is correct, per the README: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. And later in the same fi…

No, they aren’t. haasted replied to simonw ’s > "gojq does not keep the order of object keys" is a bit disappointing with > “I bet it's an artifact of Go having a randomized iteration order over maps. Getting a deterministic ordering requires extra work.” But deterministic iteration order doesn’t imply that the order of keys is kept the same. There are map implementations that keep iteration follow insertion order, b…

I don't think the distinction you're trying to make is helpful here if I've understood you correctly. A good faith interpretation of haastad's comment would be that they were thinking of "insertion order" when they said "a deterministic ordering". Even if we were being pedantic, their comment is still correct - for iteration order to be the same as input order then deterministic iteration ordering isn't sufficient (this seems to be the point you're making) but it is necessary.

Their first sentence:

> I bet it's an artifact of Go having a randomized iteration order over maps

is correct per Gojq's author [0]:

> gojq cannot implement keys_unsorted function because it uses map[string]interface{} to represent JSON object so it does not keep the order. Currently there is no plan to implement unordered map so I do not implement this function.

It would of course be possible to work around this limitation of Go's built-in map type but that's not the point. The author makes it clear that this limitation is the cause for Gojq's behaviour.

[0] https://github.com/itchyny/gojq/issues/50

Re: Gojq: Pure Go Implementation of Jq

#76

Post by nh23423fefe »

okasaki wrote: post #63

Why use a special syntax that's hard to remember when you can just use Python? I wrote a jq-like that accepts Python syntax called pq: https://github.com/dvolk/pq So you can write stuff like: $ echo '{ "US": 3, "China": 12, "UK": 1 }' | pq -c "sum(data.values())" 16

i deny the premise obviously

Re: Gojq: Pure Go Implementation of Jq

#78

Post by pbsds »

renewiltord wrote: post #8
fwip wrote: post #3

Not implementing key-sorting is a curious decision: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. I f…

Could pipe through gron and sort to resort

Gron has the same issue, as it too is written in go and randomizes key order.
Post reply on HN