Table of contents seems inspired by the famous ripgrep post from 2016: https://burntsushi.net/ripgrep/
A Faster Alternative to Jq
251–260 of 281 posts
Re: A Faster Alternative to Jq
#252Earlier quoted context omitted.
oh my god how could I have been doing this for so long and not realize that you can redirect before your binary. I knew cat was an anti-pattern, but I always thought it was so unreadable to redirect at the end
it seems smart until you accidently type >input.json and nuke the file
You can use sponge for that.
Re: A Faster Alternative to Jq
#253Earlier quoted context omitted.
Here's an easier to understand query for what you're trying to do (at least it's easier to understand for me): cat input.json | jq -r '(first | keys) as $cols | $cols, (.[] | [.[$cols[]]]) | @tsv' That whole map and from entries throws it off. It's not a good use for what you're doing. tsv expects a bunch of arrays, whereas you're getting a bunch of objects (with the header also being one) and then converting them to…
Thanks for sharing, this is much better, though I actually think it is the perfect example to explain something that is brain-slippery about jq look at $cols | $cols my brain says hmm that's a typo, clearly they meant ; instead of | because nothing is getting piped, we just have two separate statements. Surely the assignment "exhausts the pipeline" and we're only passing null downstream the pipelining has some implic…
I totally agree, it did take me a while to come to terms with the syntax of assigning variables specifically due to that pipe at the end. I guess sometimes we just have to know the quirks of the relevant tooling we use. I used to use PHP heavily in the 4 and 5 days, and kinda got used to all the quirks it had. So during reviews, I would pick up a lot of issues some of my colleagues did not.
Interestingly some things do use a semicolon in jq, specifically while, until, reduce and some others I can't remember right now.
Re: A Faster Alternative to Jq
#254Earlier quoted context omitted.
oh my god how could I have been doing this for so long and not realize that you can redirect before your binary. I knew cat was an anti-pattern, but I always thought it was so unreadable to redirect at the end
it seems smart until you accidently type >input.json and nuke the file
That sounds like a mistake which would be easily to make at the end of the line, unless you are contrasting input stream redirect against cat regardless where it's written on the line?
Re: A Faster Alternative to Jq
#255Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.
I think the big problem is it's a tool you usually reach for so rarely you never quite get the opportunity to really learn it well, so it always remains in that valley of despair where you know you should use it, but it's never intuitive or easy to use. It's not unique in that regard. 'sed' is Turing complete[1][2], but few people get farther than learning how to do a basic regex substitution. [1] https://catonmat.ne…
Yeah, sed (and friends) browbeat everyone into learning regex (which PERL then refined).
I think it might be more cognitive load than it is worth to expect everyone en masse to learn another single-line-punctuation-driven-language to perform everyday tasks with.
Re: A Faster Alternative to Jq
#256Earlier quoted context omitted.
Because the output you get can have hallucinations, which don’t happen with a deterministic tool. Furthermore, by getting the `jq` command you get something which is reusable, fast, offline, local, doesn’t send your data to a third-party, doesn’t waste a bunch of tokens, … Using an LLM to filter the data is worse in every metric.
I get that AI isn’t deterministic by definition, but IMHO it’s become the go-to response for a reason to not use AI, regardless of the use case. I’ve never seen AI “hallucinate” on basic data transformation tasks. If you tell it to convert JSON to YAML, that’s what you’re going to get. Most LLMs are probably using something like jq to do the conversion in the background anyway. AI experts say AI models don’t hallucin…
LLMs will often helpfully predict made up tokens for the content of the data fields.
For 100% of jq use cases I have the data wouldn’t fit into context. But even for the smaller things, I have never, not even once, had an LLM not mangle data that is fed into it.
Take a feed of blog posts (and select the first 50 or so just to give the model a fighting chance). I’ll give you 80% likelihood of the output being invalid JSON. And if you manage to get valid JSON out of it, the actual dates, times and text content will have changed.
Re: A Faster Alternative to Jq
#257Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.
When I need it i find that relearning the jq syntax is still faster than whatever other harebrained scheme I might come up with to solve my problem. It’s just so useful 2x a year when I really need it
Re: A Faster Alternative to Jq
#258Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.
yeah I literally just use gemini / claude to one-shot JQ queries now
I've been calling LLMs superhuman at writing `jq`. It's like you're talking directly with the JSON.
Re: A Faster Alternative to Jq
#259Having used `jq` and `yq` (which followed from the former, in spirit), I have never had to complain about performance of the _latter_ which an order of magnitude (or several) _slower_ than the former. So if there's something faster than `jq`, it's laudable that the author of the faster tool accomplished such a goal, but in the broader context I'd say the performance benefit would be required by a niche slice of the u…
> Having used `jq` and `yq` If you don't mind me asking, which yq? There's a Go variant and a Python pass-through variant, the latter also including xq and tomlq.
Indeed, thanks for spotting that, as I myself remember discovering there's at least two. Thing is, I had learned and started with Mike Farah's `yq`, not the pass-through-to-`jq` variant written in Python that's often more easily (read: system package manager) available. Both semantics and syntax are a bit different between the two.
A bit of a fun fact: there's a quote by Farah where he said that the language and semantics of the tool he was writing, didn't really "click in" until he was well into writing it :-) I myself have been on occasion pulling my hair out trying to wield `yq`'s language, there's some inconsistencies here and there which I think are related to the novel nature of the language (not novel to everyone but it's uncommon even for those well versed with e.g. SQL). `jq` suffers from similar woes, but to a lesser degree.
Re: A Faster Alternative to Jq
#260Having used `jq` and `yq` (which followed from the former, in spirit), I have never had to complain about performance of the _latter_ which an order of magnitude (or several) _slower_ than the former. So if there's something faster than `jq`, it's laudable that the author of the faster tool accomplished such a goal, but in the broader context I'd say the performance benefit would be required by a niche slice of the u…
Yeah, turns out not everyone uses these tools the way you do. Weird!
Fair enough, I deserved that :-)