Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gojq: Pure Go Implementation of Jq

github.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Gojq: Pure Go Implementation of Jq

#51

Post by nemothekid »

tialaramex wrote: post #43
chippiewill wrote: post #31

Earlier quoted context omitted.

> In some communities the reaction would have been to write a good unordered dict which would obviously be even faster Actually an ordered dictionary has improved performance over an unordered dictionary for the kinds of common Python workloads you encounter in the real world. The reason why is that the design is only incidentally ordered, the design arises from trying to improve memory efficiency and iteration speed…

The main thrust of your claim obviously can't be true and I'm not sure what confusion could lead you to believe that. Maybe it's easier to see if we're explicit about what the rules are: OrderedDict (now the Python dict) is exactly the same features as a hypothetical UnorderedDict except OrderedDict has the additional constraint that if we iterate over it we get the key/values in the order in which they were inserted…

>The main thrust of your claim obviously can't be true

It's surprising that iterating a dense array is faster than iterating a hashmap? I don't think you are parsing the parent post correctly.

If dictionaries are commonly iterated in python, then iterating an array of 100 items that fits in one cache-line will be faster than iterating a hashmap which might have 100 items in 100 cache lines.

Re: Gojq: Pure Go Implementation of Jq

#52

Post by latchkey »

simonw wrote: post #2

"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices. I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON a…

Best 3rd party Map library that I've found. https://github.com/cornelk/hashmap

Re: Gojq: Pure Go Implementation of Jq

#54

Post by gabereiser »

Groxx wrote: post #32
c2h5oh wrote: post #7

Earlier quoted context omitted.

Go actually went in the other direction for a bunch of reasons (e.g. hash collision dos) and made key order quasi-random when iterating. Small maps used to maintain order, but a change was made to randomize that so people didn't rely on that and get stung when their maps got larger: https://github.com/golang/go/issues/6719

`select{..}` cases with multiple valid channel operations also select randomly. I really like it, it helps you discover (and fix) order-dependent logic WAY earlier. Though I would really like some way to influence how long it blocks before selecting one (to simulate high load scenarios, and trigger more logical races).

you'll need an interrupt chan for that if you're in a select{..}

Re: Gojq: Pure Go Implementation of Jq

#55

Post by Lurkars »

simonw wrote: post #2

"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices. I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON a…

This. For one project, I even write a tool to reorder keys to a specific order. And of course this has no technically reason. But I used JSON here for the human readability and that non-technical people have best changes to understand and change the data. And therefore starting with id and name on top is important than with an huge array of data.

Re: Gojq: Pure Go Implementation of Jq

#56

Post by silon42 »

esprehn wrote: post #27
vips7L wrote: post #24

Earlier quoted context omitted.

Does Go not have more than one Map implementation in the standard library?

It does not. Maps are not even a real interface you can implement, it's compiler magic encoded in the language spec: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem... This is all fallout of not having generics.

So, golang is no longer suitable to nicely process json documents.

Re: Gojq: Pure Go Implementation of Jq

#58

Post by hoppla »

simonw wrote: post #6
haasted wrote: post #5

Earlier quoted context omitted.

I bet it's an artifact of Go having a randomized iteration order over maps [0]. Getting a deterministic ordering requires extra work. [0] https://stackoverflow.com/questions/9619479/go-what-determin...

I used to have the exact same problem with Python, until Python 3.7 made maintaining sort order a feature of the language: https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/

This burnt me when I wrote an algorithm. I depended on the order of keys in dicts as it allowed me reference the value both by index and key.

I wrote the code in python 3.7+, and ended up spending a good amount of time debugging it when I ran it in a earlier python version.

Re: Gojq: Pure Go Implementation of Jq

#59

Post by lemagedurage »

nerdponx wrote: post #45
chippiewill wrote: post #31

Earlier quoted context omitted.

> In some communities the reaction would have been to write a good unordered dict which would obviously be even faster Actually an ordered dictionary has improved performance over an unordered dictionary for the kinds of common Python workloads you encounter in the real world. The reason why is that the design is only incidentally ordered, the design arises from trying to improve memory efficiency and iteration speed…

The problem with Python here is that CPython is not only the reference implementation but the de-facto specification. So dicts are still "supposed to be" unordered collections, but now dicts must also preserve insertion order as per the docs and the reference implementation, so now all alternative implementations must also conform to this even if it doesn't make sense for them to conform to it, or they must specifica…

Since Python 3.7 preservering insertion-order is part of the language specification.

"the insertion-order preservation nature of dict objects has been declared to be an official part of the Python language spec."

https://docs.python.org/3/whatsnew/3.7.html

Re: Gojq: Pure Go Implementation of Jq

#60

Post by tialaramex »

nemothekid wrote: post #51
tialaramex wrote: post #43

Earlier quoted context omitted.

The main thrust of your claim obviously can't be true and I'm not sure what confusion could lead you to believe that. Maybe it's easier to see if we're explicit about what the rules are: OrderedDict (now the Python dict) is exactly the same features as a hypothetical UnorderedDict except OrderedDict has the additional constraint that if we iterate over it we get the key/values in the order in which they were inserted…

> The main thrust of your claim obviously can't be true It's surprising that iterating a dense array is faster than iterating a hashmap? I don't think you are parsing the parent post correctly. If dictionaries are commonly iterated in python, then iterating an array of 100 items that fits in one cache-line will be faster than iterating a hashmap which might have 100 items in 100 cache lines.

The claim was that: "Actually an ordered dictionary has improved performance over an unordered dictionary"

Having a dense array is not, as it seems both you and chippiewill imagine, somehow a unique property of ordered dictionaries. An unordered dictionary is free to use exactly the same implementation detail.

The choice to preserve order is in addition to using dense arrays. The OrderedDict must use tombstones in order to preserve ordering, and then periodically rewrite the entire dense array to remove tombstones, while a hypothetical UnorderedDict needn't worry because it isn't trying to preserve ordering so it will be faster here despite also having the dense arrays.

"iterating an array of 100 items that fits in one cache-line will be faster"

On today's hardware a cache line is 64 bytes, so fitting 100 "items" (each 3x 64-bit values, so typically total 2400 bytes with today's Python implementation) in a cache line would not be possible. A rather less impressive "almost three" items fit in a cache line.

But to be sure the dense array is faster for this operation, the problem is that that's not an optimisation as a result of being ordered. It's just an implementation choice and the UnorderedDict is free to make the same choice.

Post reply on HN