Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gojq: Pure Go Implementation of Jq

github.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Gojq: Pure Go Implementation of Jq

#41

Post by moharoune »

edsiper2 wrote: post #39

Naming is hard, but please, do not repeat the mistake of many OSS project in the last 20 years calling each project by prefixing the name with the stack/environment involved. Now a "trending" language can catch the attention, but tomorrow?.. maybe. So the value proposition and starting from it name should be different (if you want adoption). For my use case, for a rewrite of jq I would expect one thing only: higher p…

I'd also expect higher performance for a rewrite of jq or, for that matter, any other tool that works as expected and being used for a long time.

Re: Gojq: Pure Go Implementation of Jq

#42

Post by tialaramex »

mdaniel wrote: post #19
eropple wrote: post #4

Earlier quoted context omitted.

I agree with you that there's value to sorted keys from a presentational standpoint (we are not beep-boop robots, humans have to read this stuff too), but now there also exists a JSON canonicalization RFC that tools can/should follow (with all the usual caveats about canonicalization being fraught): https://www.rfc-editor.org/rfc/rfc8785

I guess "Informational" is better than /dev/null, but unless everyone adopts it doesn't that run the risk of it just being My Favorite Canonicalization™? Either way, I'm guessing if the gojq author has that much heartburn about implementing --sort-keys, --canonical is just absolutely off the table :-(

> unless everyone adopts it doesn't that run the risk of it just being My Favorite Canonicalization

That's true regardless. The IETF has no enforcement arm. Even if people expend the effort to agree a standard, and make whatever signs and follow the rituals, if nobody implements it then de facto that isn't the standard after all.

Re: Gojq: Pure Go Implementation of Jq

#43

Post by tialaramex »

chippiewill wrote: post #31
tialaramex wrote: post #13

Earlier quoted context omitted.

Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. It's like if you did such a bad job making a drag racer that the street legal model of the same car was substantially faster over a quarter mile despite also having much better handling and reliability. In some communities the reaction would have been to write a good unordered d…

> In some communities the reaction would have been to write a good unordered dict which would obviously be even faster Actually an ordered dictionary has improved performance over an unordered dictionary for the kinds of common Python workloads you encounter in the real world. The reason why is that the design is only incidentally ordered, the design arises from trying to improve memory efficiency and iteration speed…

The main thrust of your claim obviously can't be true and I'm not sure what confusion could lead you to believe that.

Maybe it's easier to see if we're explicit about what the rules are: OrderedDict (now the Python dict) is exactly the same features as a hypothetical UnorderedDict except OrderedDict has the additional constraint that if we iterate over it we get the key/values in the order in which they were inserted, while UnorderedDict can do as it pleases here.

This means OrderedDict is a valid implementation of UnorderedDict. So, necessarily OrderedDict does not have, as you claim, "improved performance over an unordered dictionary". At the very worst it's break even and performance is identical. This is why it's remarkable that Python's previous dict was worse.

But, that's a pretty degenerate case, we can also see that after deletion OrderedDict must use some resources ensuring the ordering constraint is kept. An UnorderedDict needn't do that, and we can definitely do better than OrdererDict.

Re: Gojq: Pure Go Implementation of Jq

#44

Post by jeffbee »

moharoune wrote: post #41
edsiper2 wrote: post #39

Naming is hard, but please, do not repeat the mistake of many OSS project in the last 20 years calling each project by prefixing the name with the stack/environment involved. Now a "trending" language can catch the attention, but tomorrow?.. maybe. So the value proposition and starting from it name should be different (if you want adoption). For my use case, for a rewrite of jq I would expect one thing only: higher p…

I'd also expect higher performance for a rewrite of jq or, for that matter, any other tool that works as expected and being used for a long time.

Last time I profiled jq in my particular use case - querying large GeoJSON files - I discovered it spent practically all of its CPU in assert, and it went a lot faster when built with -DNDEBUG, but since I could not rule out that some of its asserts have side effects I went back to the upstream package.

I think beating the performance of jq would be very easy for anyone who set out with that as a goal. It also has its own internal strtod and dtoa which are easily beaten by ryu or C++'s from/to_chars, so I would start there after dumping the weird asserts.

Re: Gojq: Pure Go Implementation of Jq

#45

Post by nerdponx »

chippiewill wrote: post #31
tialaramex wrote: post #13

Earlier quoted context omitted.

Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. It's like if you did such a bad job making a drag racer that the street legal model of the same car was substantially faster over a quarter mile despite also having much better handling and reliability. In some communities the reaction would have been to write a good unordered d…

> In some communities the reaction would have been to write a good unordered dict which would obviously be even faster Actually an ordered dictionary has improved performance over an unordered dictionary for the kinds of common Python workloads you encounter in the real world. The reason why is that the design is only incidentally ordered, the design arises from trying to improve memory efficiency and iteration speed…

The problem with Python here is that CPython is not only the reference implementation but the de-facto specification. So dicts are still "supposed to be" unordered collections, but now dicts must also preserve insertion order as per the docs and the reference implementation, so now all alternative implementations must also conform to this even if it doesn't make sense for them to conform to it, or they must specifically choose to be non-comformant on this point.

Of course in this case, the order-preserving optimization was actually first implemented by an alternative implementation (PyPY), but I don't think that changes the issue.

Re: Gojq: Pure Go Implementation of Jq

#46

Post by hxtk »

xmonkee wrote: post #40
esprehn wrote: post #27

Earlier quoted context omitted.

It does not. Maps are not even a real interface you can implement, it's compiler magic encoded in the language spec: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem... This is all fallout of not having generics.

I fucking hate this so much. Honestly Go isn't a bad language, but I dunno why these kind of things just piss me off.

It seems as though a lot of people view it as hypocritical, e.g., generics for me but not for thee (dated example since there are now generics for everyone).

The fact that they needed to make a map a part of the language in order to allow it to be generic and statically-typed proves that generics are useful and should therefore have been a language feature much earlier than they became one.

There are a variety of things that the standard library or compiler deal with using weird workarounds that seem to indicate missing language features.

The thing is, the features are only "missing" if the language is designed to do the things those features permit. So the counterargument is that Go is a very opinionated language designed to do solve a few classes of problem very easily, like writing database-backed web services, and the reason the standard library or compiler teams have to do weird hacks at times is because Go wasn't made for writing those things, and designing to make those use cases easy would pollute the language from the perspective of someone using it for its intended purpose.

Re: Gojq: Pure Go Implementation of Jq

#47

Post by comex »

xmonkee wrote: post #40
esprehn wrote: post #27

Earlier quoted context omitted.

It does not. Maps are not even a real interface you can implement, it's compiler magic encoded in the language spec: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem... This is all fallout of not having generics.

I fucking hate this so much. Honestly Go isn't a bad language, but I dunno why these kind of things just piss me off.

Well, keep in mind that it’s out of date; Go does have generics now.

Re: Gojq: Pure Go Implementation of Jq

#48

Post by oorza »

hxtk wrote: post #46
xmonkee wrote: post #40

Earlier quoted context omitted.

I fucking hate this so much. Honestly Go isn't a bad language, but I dunno why these kind of things just piss me off.

It seems as though a lot of people view it as hypocritical, e.g., generics for me but not for thee (dated example since there are now generics for everyone). The fact that they needed to make a map a part of the language in order to allow it to be generic and statically-typed proves that generics are useful and should therefore have been a language feature much earlier than they became one. There are a variety of thi…

It's really hard to take those arguments without a whole serving of salt because the things that it's ostensibly good at handling really aren't that much easier. Why is (un)marshalling data in a type safe way so damn hard in Go? Why does doing the same thing over and over and over and over never get easier? (Because the language lacks high level abstractions.

I used Go extensively at my last job and I was left feeling that there were pretty much always better choices. If you care about developer velocity with your more unskilled engineers, Go is a bad choice for a multitude of reasons. If you're going to need the performance over something else, the JVM is right there, and so too is Rust.

Re: Gojq: Pure Go Implementation of Jq

#49

Post by ZephyrBlu »

esprehn wrote: post #27
vips7L wrote: post #24

Earlier quoted context omitted.

Does Go not have more than one Map implementation in the standard library?

It does not. Maps are not even a real interface you can implement, it's compiler magic encoded in the language spec: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem... This is all fallout of not having generics.

I would have never expected that a language (Especially a compiled one) does that kind of fuckery behind the scenes.

Re: Gojq: Pure Go Implementation of Jq

#50

Post by aarchi »

lapser wrote: post #10

I have actually fully replaced my jq installation with gojq (including an `ln -s gojq jq`) for a few years, and no script has broken so far. I'm super impressed by the jq compatibility. If you are going down this route, do be careful with performance. I don't know which is more performant as I've never really had to work with large data sets, but I can't help but feel jq will be faster than gojq in such case. I have…

To see if gojq works even with complex jq programs, I tested it on my wsjq[0] Whitespace language interpreter, which uses most of the advanced jq features. It impressively appears to support the full jq language, though I uncovered a bug[1] in gojq.

gojq's arbitrary-precision integer support will be useful (jq just uses 64-bit floating-point), though I suspect it will have performance regressions, since it uses math/big, instead of GMP.

[0]: https://github.com/andrewarchi/wsjq

[1]: https://github.com/itchyny/gojq/issues/186

Post reply on HN