Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Faster Alternative to Jq

micahkepe.com

Post reply on HN
Print view View original

231–240 of 281 posts

Re: A Faster Alternative to Jq

#231

Post by jasomill »

marginalia_nu wrote: post #206
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

I think the big problem is it's a tool you usually reach for so rarely you never quite get the opportunity to really learn it well, so it always remains in that valley of despair where you know you should use it, but it's never intuitive or easy to use. It's not unique in that regard. 'sed' is Turing complete[1][2], but few people get farther than learning how to do a basic regex substitution. [1] https://catonmat.ne…

I was just going to say, jq is like sed in that I only use 1% of it 99% of the time, but unlike sed in that I'm not aware of any clearly better if less ubiquitous alternatives to the 1% (e.g., Perl or ripgrep for simple regex substitutions in pipelines because better regex dialects).

Closest I've come, if you're willing to overlook its verbosity and (lack of) speed, is actually PowerShell, if only because it's a bit nicer than Python or JavaScript for interactive use.

Re: A Faster Alternative to Jq

#232

Post by dcre »

regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

Funny that everyone is linking the tools they wrote for themselves to deal with this problem. I am no exception. I wrote one that just lets you write JavaScript. Imagine my surprise that this extremely naive implementation was faster than jq, even on large files.

    $ cat package.json | dq 'Object.keys(data).slice(0, 5)'
    [ "name", "type", "version", "scripts", "dependencies" ]
https://crespo.business/posts/dq-its-just-js/

Re: A Faster Alternative to Jq

#233

Post by dcre »

LgWoodenBadger wrote: post #230
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

I completely agree. I much prefer leveraging actual javascript to get what I need instead of spending time trying to fumble my way through jq syntax.

Check this out: https://crespo.business/posts/dq-its-just-js/

You don't have to use my implementation, you could easily write your own.

Re: A Faster Alternative to Jq

#234

Post by sramsay »

1a527dd5 wrote: post #43

I appreciate performance as much as the next person; but I see this endless battle to measure things in ns/us/ms as performative. Sure there are 0.000001% edge cases where that MIGHT be the next big bottleneck. I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart. 9/10 whatever tooling you are using now will be perfectly fine. Example; I use grep a…

I absolutely understand what you're saying. It makes complete sense. But I will never, ever shake the sense that software that isn't as fast as possible is offensive, immoral, delinquent -- the result of sloth, lassitude, lack of imagination, and a general hostility toward our noble Art.

"Fast enough" will always bug me. "Still ahead of network latency" will always sound like the dog ate your homework. I understand the perils of premature optimization, but not a refusal to optimize.

And I doubt I'm alone.

Re: A Faster Alternative to Jq

#235

Post by jasomill »

norenh wrote: post #210
marxisttemp wrote: post #75

Many Useless Uses of cat in this documentation. You never need to do `cat file | foo`, you can just do `<file foo`. cat is for concatenating inputs, you never need it for a single input.

As someone who worked with Unix/Linux and command line arguments for 30 years and still "abuse" cat like the documentation, I regularly hear this complaint. Yes, "cmd At least for me, the first variant comes more naturally and is quicker to follow (in most cases), so unless it is performance sensitive that is what I end up with (and cat is insanely fast for most cases).

If left-to-right is your main concern, observe that the post you replied to uses
which is equivalent to

  command

Re: A Faster Alternative to Jq

#236

Post by attentive »

d35007 wrote: post #160
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them. I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think…

> I dream of a world in which all CLI tools produce and consume JSON and we use jq to glue them together.

that world exists and mature (powershell)

Re: A Faster Alternative to Jq

#237

Post by dannyobrien »

NSPG911 wrote: post #142
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

I also genuinely hate using jq. It is one of the only things that I rely heavily on AI.

I use the llm-jq plugin for Simon Willison's `llm` command line frontend for this: https://github.com/simonw/llm-jq

Re: A Faster Alternative to Jq

#238

Post by attentive »

randusername wrote: post #227
nh23423fefe wrote: post #220

Earlier quoted context omitted.

oh my god how could I have been doing this for so long and not realize that you can redirect before your binary. I knew cat was an anti-pattern, but I always thought it was so unreadable to redirect at the end

it seems smart until you accidently type >input.json and nuke the file

Re: A Faster Alternative to Jq

#239

Post by rzzzt »

d35007 wrote: post #160
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them. I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think…

I'm often having trouble with figuring out in advance what the end result will be when processing an input array: an array of mapped objects or a series of self-contained JSON objects? Why? Which one is better? What if I would like to filter out some of the elements as part of the operation?

Re: A Faster Alternative to Jq

#240

Post by landr0id »

loxias wrote: post #208
n_e wrote: post #11

Earlier quoted context omitted.

I process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so slow that I write a single-use node or rust script instead.

I would love, _love_ to know more about your data formats, your tools, what the JSON looks like, basically as much as you're willing to share. :) For about a month now I've been working on a suite of tools for dealing with JSON specifically written for the imagined audience of "for people who like CLIs or TUIs and have to deal with PILES AND PILES of JSON and care deeply about performance". For me, I've been writing…

I maintain some tools for the videogame World of Warships. The developer has a file called GameParams.bin which is Python-pickled data (their scripting language is Python).

Working with this is pretty painful, so I convert the Pickled structure to other formats including JSON.

The file has always been prettified around ~500MB but as of recently expands to about 3GB I think because they’ve added extra regional parameters.

The file inflates to a large size because Pickle refcounts objects for deduping, whereas obviously that’s lost in JSON.

I care about speed and tools not choking on the large inputs so I use jaq for querying and instruction LLMs operating on the data to do the same.

Post reply on HN