Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Faster Alternative to Jq

micahkepe.com

Post reply on HN
Print view View original

271–280 of 281 posts

Re: A Faster Alternative to Jq

#271

Post by xnx »

d35007 wrote: post #160
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them. I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think…

It's extra complicated under Windows because of issues escaping/wrapping quotes "" and pipes ^|.

Re: A Faster Alternative to Jq

#273

Post by Self-Perfection »

I think that in most cases jq is launched to extract value from relatively small JSON document, for which raw parsing speed is not affect much. jq is just really slow to start. Version 1.6 was especially abysmally slow to start, 10x times slower than 1.5:

https://github.com/jqlang/jq/issues/1826

So any replacement candidate should also benchmark like hyperfine "jq .a Also please just use jshon if you need to just extract specific value from some small JSON. jshon uses way less resources by any conceivable metric.

Re: A Faster Alternative to Jq

#275

Post by ndyg »

ifh-hn wrote: post #19

I learned a number of data processing cli tools: jq, mlr, htmlq, xsv, yq, etc; to name a few. Not to the level of completing advent of code or anything, but good enough for my day to day usage. It was never ending with the amount of formats I needed to extract data from, and the different syntax's. All that changed when I found nushell though, its replaced all of these tools for me. One syntax for everything, breath…

Something I find myself saying a lot, Nushell is a better `jq` than `jq`

Re: A Faster Alternative to Jq

#276

Post by alwillis »

tkclough wrote: post #218
alwillis wrote: post #204

Earlier quoted context omitted.

I get that AI isn’t deterministic by definition, but IMHO it’s become the go-to response for a reason to not use AI, regardless of the use case. I’ve never seen AI “hallucinate” on basic data transformation tasks. If you tell it to convert JSON to YAML, that’s what you’re going to get. Most LLMs are probably using something like jq to do the conversion in the background anyway. AI experts say AI models don’t hallucin…

Just because you haven't seen it hallucinate on these tasks doesn't mean it can't. When I'm deciding what tool to use, my question is "does this need AI?", not "could AI solve this?" There's plenty of cases where its hard to write a deterministic script to do something, but if there is a deterministic option, why would you choose something that might give you the wrong answer? It's also more expensive. The jq script…

> but if there is a deterministic option, why would you choose something that might give you the wrong answer?

Claude Code can use jq if it's installed on your system. Also, the data transformation is usually part of a larger workflow where an LLM is being used. And honestly, Claude is going to know jq better than 95% of developers who use it. jq can do a lot of things but it’s not the most intuitive tool to learn.

An obvious best practice is to have the LLM use existing tools to confirm the correctness of its output.

Re: A Faster Alternative to Jq

#277

Post by alwillis »

djhn wrote: post #256
alwillis wrote: post #204

Earlier quoted context omitted.

I get that AI isn’t deterministic by definition, but IMHO it’s become the go-to response for a reason to not use AI, regardless of the use case. I’ve never seen AI “hallucinate” on basic data transformation tasks. If you tell it to convert JSON to YAML, that’s what you’re going to get. Most LLMs are probably using something like jq to do the conversion in the background anyway. AI experts say AI models don’t hallucin…

LLMs will often helpfully predict made up tokens for the content of the data fields. For 100% of jq use cases I have the data wouldn’t fit into context. But even for the smaller things, I have never, not even once, had an LLM not mangle data that is fed into it. Take a feed of blog posts (and select the first 50 or so just to give the model a fighting chance). I’ll give you 80% likelihood of the output being invalid…

I’ll have to give this a shot.

One possibility: Claude Code subagents get their own 1 million token context window; should be better with large JSON files vs. having everything in the same context window.

Re: A Faster Alternative to Jq

#278

Post by rennokki »

eevmanu wrote: post #178
rennokki wrote: post #71

Earlier quoted context omitted.

> Uses jq for TB json files > Hadoop: bro > Spark: bro > hive: bro > data team: bro

made me remember this article https://adamdrake.com/command-line-tools-can-be-235x-faster-... > Command-line Tools can be 235x Faster than your Hadoop Cluster (2014) Conclusion: Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.

This article is good for new programmers to understand why certain solutions are better at scale, there is no silver bullet. And also, this is from 2014, and the dataset is The discussion we had here was involving TB of data, so I'm curious how this is faster with CLIs rather than parallel processing...

Re: A Faster Alternative to Jq

#279

Post by rennokki »

f311a wrote: post #112
rennokki wrote: post #71

Earlier quoted context omitted.

> Uses jq for TB json files > Hadoop: bro > Spark: bro > hive: bro > data team: bro

JQ is very convenient, even if your files are more than 100GB. I often need to extract one field from huge JSON line files, I just pipe jq to it to get results. It's slower, but implementing proper data processing will take more time.

More than 100GB can be 101GB, 500GB or 1TB+. I was speaking about 1TB+ files. I'm not sure you can get it faster unless you have a parallel processor.
Post reply on HN