Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Faster Alternative to Jq

micahkepe.com

Post reply on HN
Print view View original

211–220 of 281 posts

Re: A Faster Alternative to Jq

#211

Post by jiehong »

micahkepe wrote: post #198
jiehong wrote: post #41

First of all, congratulations! Nice tool! Second, some comments on the presentation: the horizontal violin graphs are nice, but all tools have the same colours, and so it's just hard to even spot where jsongrep is. I'd recommend grouping by tool and colour coding it. Besides, jq itself isn't in the graphs at all (but the title of the post made me think it would be!). Last, xLarge is a 190MiB file. I was surprised by…

Hey thank you! OP here, yes I was struggling to find large enough documents to run the benchmarks on, the range currently on the benchmark data is ~106 B - ~190MB, which I think covers the majority of quick task workloads, but would love to have large documents, if there's an public ones you can thinking of I'd like to know!

The US government tend to offer big public json document [0], such as crime rates [1], or others.

[0]: https://catalog.data.gov/dataset/?res_format=JSON

[1]: https://catalog.data.gov/dataset/crimes-2001-to-present

Re: A Faster Alternative to Jq

#212

Post by randusername »

figmert wrote: post #209
randusername wrote: post #186

Earlier quoted context omitted.

I'm not GP, I use jq all the time, but I each time I use it I feel like I'm still a beginner because I don't get where I want to go on the first several attempts. Great tool, but IMO it is more intuitive to JSON people that want a CLI tool than CLI people that want a JSON tool. In other words, I have my own preconceptions about how piping should work on the whole thing, not iterating, and it always trips me up. Here'…

Here's an easier to understand query for what you're trying to do (at least it's easier to understand for me): cat input.json | jq -r '(first | keys) as $cols | $cols, (.[] | [.[$cols[]]]) | @tsv' That whole map and from entries throws it off. It's not a good use for what you're doing. tsv expects a bunch of arrays, whereas you're getting a bunch of objects (with the header also being one) and then converting them to…

Thanks for sharing, this is much better, though I actually think it is the perfect example to explain something that is brain-slippery about jq

look at $cols | $cols

my brain says hmm that's a typo, clearly they meant ; instead of | because nothing is getting piped, we just have two separate statements. Surely the assignment "exhausts the pipeline" and we're only passing null downstream

the pipelining has some implicit contextual stuff going on that I have to arrive at by trial and error each time since it doesn't fit in my worldview while I'm doing other shell stuff

Re: A Faster Alternative to Jq

#213

Post by keysersoze33 »

carlmr wrote: post #31
keysersoze33 wrote: post #9

I was a bit skeptical at first, but after reading more into jsongrep, it's actually very good. Only did a very quick test just now, and after stumbling over slightly different syntax to jq, am actually quite impressed. Give it a try

What were your syntax stumbling blocks? I must be honest I've used jq enough but can never remember the syntax. It's one of the worst things about jq IMO (not the speed, even though I'm a fan of speedups). There's something ungrokkable about that syntax for me.

Just the basic things, like viewing the complete json (with syntax highlighting) to then determine the filter, that is '.' becomes '**'

Re: A Faster Alternative to Jq

#214

Post by jeffbee »

Jenk wrote: post #106

I switched to Jaq[0] a while back for the 'correctness' sake rather than performance. But Jaq also claims to be more performant than jq. [0]: https://github.com/01mf02/jaq

I keep an eye on jaq, but there are some holes in the story. jaq 3.0 is faster than Linux distro builds of jq, but jq built correctly is faster than jaq. As far as I can tell the performance reputation of jq is caused by bad distro packaging.

Re: A Faster Alternative to Jq

#215

Post by epr »

regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

To fix this I recently made myself a tiny tool I called jtree that recursively walks json, spitting out one line per leaf. Each line is the jq selector and leaf value separated by "=".

No more fiddling around trying to figure out the damn selector by trying to track the indentation level across a huge file. Also easy to pipe into fzf, then split on "=", trim, then pass to jq

Re: A Faster Alternative to Jq

#216

Post by iamjackg »

epr wrote: post #215
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

To fix this I recently made myself a tiny tool I called jtree that recursively walks json, spitting out one line per leaf. Each line is the jq selector and leaf value separated by "=". No more fiddling around trying to figure out the damn selector by trying to track the indentation level across a huge file. Also easy to pipe into fzf, then split on "=", trim, then pass to jq

You might like https://github.com/tomnomnom/gron

Re: A Faster Alternative to Jq

#217

Post by phillipcarter »

1a527dd5 wrote: post #43

I appreciate performance as much as the next person; but I see this endless battle to measure things in ns/us/ms as performative. Sure there are 0.000001% edge cases where that MIGHT be the next big bottleneck. I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart. 9/10 whatever tooling you are using now will be perfectly fine. Example; I use grep a…

I agree for some things, but not for tools or "micro-software" like jq that can get called a LOT in an automated process. Every order of magnitude saved for the latter category can be meaningful.

Re: A Faster Alternative to Jq

#218

Post by tkclough »

alwillis wrote: post #204
latexr wrote: post #156

Earlier quoted context omitted.

Because the output you get can have hallucinations, which don’t happen with a deterministic tool. Furthermore, by getting the `jq` command you get something which is reusable, fast, offline, local, doesn’t send your data to a third-party, doesn’t waste a bunch of tokens, … Using an LLM to filter the data is worse in every metric.

I get that AI isn’t deterministic by definition, but IMHO it’s become the go-to response for a reason to not use AI, regardless of the use case. I’ve never seen AI “hallucinate” on basic data transformation tasks. If you tell it to convert JSON to YAML, that’s what you’re going to get. Most LLMs are probably using something like jq to do the conversion in the background anyway. AI experts say AI models don’t hallucin…

Just because you haven't seen it hallucinate on these tasks doesn't mean it can't.

When I'm deciding what tool to use, my question is "does this need AI?", not "could AI solve this?" There's plenty of cases where its hard to write a deterministic script to do something, but if there is a deterministic option, why would you choose something that might give you the wrong answer? It's also more expensive.

The jq script or other script that an LLM generates is way easier to spot check than the output if you ask it to transform the data directly, and you can reuse it.

Re: A Faster Alternative to Jq

#219

Post by noisy_boy »

alsetmusic wrote: post #151
j1elo wrote: post #76

Earlier quoted context omitted.

Whenever you have this kind of impressions on some development, here are my 2 cents: just think "I'm not the target audience". And that's fine. The difference between 2ms and 0.2ms might sound unneeded, or even silly to you. But somebody, somewhere, is doing stream processing of TB-sized JSON objects, and they will care. These news are for them.

I remember when I was coming up on the command line and I'd browse the forums at unix.com. Someone would ask how to do a thing and CFAJohnson would come in with a far less readable solution that was more performative (probably replacing calls to external tools with Bash internals, but I didn't know enough then to speak intelligently about it now). People would say, "Why use this when it's harder to read and only save…

Wow that takes me back. I used to lurk on unix.com when I was starting with bash and perl and would see CFAJohnson's terse one-liners all the time. I enjoyed trying my own approaches to compare performance, conciseness and readability - mainly for learning. Some of the awk stuff was quite illuminating in my understanding of how powerful awk could be. I remember trying different approaches to process large files at first with awk and then with Perl. Then we discovered Oracle's external tables which turned out to be clear winner. We have a lot more options now with fantastic performance.

Re: A Faster Alternative to Jq

#220

Post by nh23423fefe »

randusername wrote: post #186
d35007 wrote: post #160

Earlier quoted context omitted.

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them. I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think…

I'm not GP, I use jq all the time, but I each time I use it I feel like I'm still a beginner because I don't get where I want to go on the first several attempts. Great tool, but IMO it is more intuitive to JSON people that want a CLI tool than CLI people that want a JSON tool. In other words, I have my own preconceptions about how piping should work on the whole thing, not iterating, and it always trips me up. Here'…
Post reply on HN