Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gojq: Pure Go Implementation of Jq

github.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Gojq: Pure Go Implementation of Jq

#31

Post by chippiewill »

tialaramex wrote: post #13
c2h5oh wrote: post #7

Earlier quoted context omitted.

Go actually went in the other direction for a bunch of reasons (e.g. hash collision dos) and made key order quasi-random when iterating. Small maps used to maintain order, but a change was made to randomize that so people didn't rely on that and get stung when their maps got larger: https://github.com/golang/go/issues/6719

Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. It's like if you did such a bad job making a drag racer that the street legal model of the same car was substantially faster over a quarter mile despite also having much better handling and reliability. In some communities the reaction would have been to write a good unordered d…

> In some communities the reaction would have been to write a good unordered dict which would obviously be even faster

Actually an ordered dictionary has improved performance over an unordered dictionary for the kinds of common Python workloads you encounter in the real world. The reason why is that the design is only incidentally ordered, the design arises from trying to improve memory efficiency and iteration speed. The dict ends up ordered because they stash the real k/v pairs in a regular array which is indexed by the hash table, populating the array is most efficient in insertion order. For pure "unordered map" type operations the newer implementation is actually a tiny bit slower.

Re: Gojq: Pure Go Implementation of Jq

#32

Post by Groxx »

c2h5oh wrote: post #7
simonw wrote: post #6

Earlier quoted context omitted.

I used to have the exact same problem with Python, until Python 3.7 made maintaining sort order a feature of the language: https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/

Go actually went in the other direction for a bunch of reasons (e.g. hash collision dos) and made key order quasi-random when iterating. Small maps used to maintain order, but a change was made to randomize that so people didn't rely on that and get stung when their maps got larger: https://github.com/golang/go/issues/6719

`select{..}` cases with multiple valid channel operations also select randomly.

I really like it, it helps you discover (and fix) order-dependent logic WAY earlier. Though I would really like some way to influence how long it blocks before selecting one (to simulate high load scenarios, and trigger more logical races).

Re: Gojq: Pure Go Implementation of Jq

#33

Post by 41b696ef1113 »

simonw wrote: post #2

"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices. I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON a…

It does not even provide a `--sort-keys` option. That's like 90% of the the reason I ever lean on jq - to standardize API output for my human brain.

Re: Gojq: Pure Go Implementation of Jq

#34

Post by hyperpallium2 »

simonw wrote: post #2

"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices. I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON a…

jq used to do this, but changed to preserve key order.

Re: Gojq: Pure Go Implementation of Jq

#35

Post by hyperpallium2 »

lapser wrote: post #10

I have actually fully replaced my jq installation with gojq (including an `ln -s gojq jq`) for a few years, and no script has broken so far. I'm super impressed by the jq compatibility. If you are going down this route, do be careful with performance. I don't know which is more performant as I've never really had to work with large data sets, but I can't help but feel jq will be faster than gojq in such case. I have…

jq uses bison (gnu's yacc), which is a nightmare for error diagnosis. Additionally, the founder (though brilliant - or maybe because brilliant) wouldn't accept improvements in error reporting.

Re: Gojq: Pure Go Implementation of Jq

#38

Post by adgjlsfhk1 »

aaronbee wrote: post #21
alecthomas wrote: post #16

Earlier quoted context omitted.

> Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. I've never heard that before and it would be really surprising, given that Python's builtin dict is used for everything from local symbol to object field lookup. Do you have more information?

Here’s a description of the new map implementation and why it’s more efficient: https://www.pypy.org/posts/2015/01/faster-more-memory-effici...

Note that this applies more for python than efficient languages. In python, objects are big, and require an indirection. In faster languages, many objects can be smaller than a pointer and stored inline. As such, dictionaries that have vectorized lookups generally can be made faster.

Re: Gojq: Pure Go Implementation of Jq

#39

Post by edsiper2 »

Naming is hard, but please, do not repeat the mistake of many OSS project in the last 20 years calling each project by prefixing the name with the stack/environment involved.

Now a "trending" language can catch the attention, but tomorrow?.. maybe. So the value proposition and starting from it name should be different (if you want adoption).

For my use case, for a rewrite of jq I would expect one thing only: higher performance... show the numbers ;)

Re: Gojq: Pure Go Implementation of Jq

#40

Post by xmonkee »

esprehn wrote: post #27
vips7L wrote: post #24

Earlier quoted context omitted.

Does Go not have more than one Map implementation in the standard library?

It does not. Maps are not even a real interface you can implement, it's compiler magic encoded in the language spec: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem... This is all fallout of not having generics.

I fucking hate this so much. Honestly Go isn't a bad language, but I dunno why these kind of things just piss me off.
Post reply on HN