Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Faster Alternative to Jq

micahkepe.com

Post reply on HN
Print view View original

241–250 of 281 posts

Re: A Faster Alternative to Jq

#241

Post by iLemming »

regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

It's because .json itself has so much useless cruft it's often annoying to deal with. I am forever indebted for younger self forcing me to learn Clojure. Most of the time I choose not even bother with JSON anymore - EDN semantically so much cleaner - it's almost twice compact (yet lossless), it's far more readable (quotes and commas are optional), and easier to work with structurally. These days I'd use borkdude/jet or babashka and then deal with data in Clojure REPL - there I can inspect it from all sorts of angles, it's far easier to group, sort, slice, dice, map and filter through it. One can even easily visualize the data using djblue/portal. Why most people strangulate themselves with confusing jq operators unnecessarily, I would never understand. Clojure is not that hard, maybe learn some basics, it comes handy a lot. Even when your team doesn't have any Clojure code.

Re: A Faster Alternative to Jq

#242

Post by otterley »

dcre wrote: post #232
regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

Funny that everyone is linking the tools they wrote for themselves to deal with this problem. I am no exception. I wrote one that just lets you write JavaScript. Imagine my surprise that this extremely naive implementation was faster than jq, even on large files. $ cat package.json | dq 'Object.keys(data).slice(0, 5)' [ "name", "type", "version", "scripts", "dependencies" ] https://crespo.business/posts/dq-its-just-j…

Love it!

Re: A Faster Alternative to Jq

#243

Post by lbj »

j1elo wrote: post #76
1a527dd5 wrote: post #43

I appreciate performance as much as the next person; but I see this endless battle to measure things in ns/us/ms as performative. Sure there are 0.000001% edge cases where that MIGHT be the next big bottleneck. I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart. 9/10 whatever tooling you are using now will be perfectly fine. Example; I use grep a…

Whenever you have this kind of impressions on some development, here are my 2 cents: just think "I'm not the target audience". And that's fine. The difference between 2ms and 0.2ms might sound unneeded, or even silly to you. But somebody, somewhere, is doing stream processing of TB-sized JSON objects, and they will care. These news are for them.

That's a really great perspective. Thanks for sharing!

Re: A Faster Alternative to Jq

#245

Post by firesteelrain »

randusername wrote: post #186
d35007 wrote: post #160

Earlier quoted context omitted.

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them. I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think…

I'm not GP, I use jq all the time, but I each time I use it I feel like I'm still a beginner because I don't get where I want to go on the first several attempts. Great tool, but IMO it is more intuitive to JSON people that want a CLI tool than CLI people that want a JSON tool. In other words, I have my own preconceptions about how piping should work on the whole thing, not iterating, and it always trips me up. Here'…

Trying to make a generic pipeline for json arrays because you don’t know the field names?

Re: A Faster Alternative to Jq

#246

Post by firesteelrain »

alsetmusic wrote: post #151
j1elo wrote: post #76

Earlier quoted context omitted.

Whenever you have this kind of impressions on some development, here are my 2 cents: just think "I'm not the target audience". And that's fine. The difference between 2ms and 0.2ms might sound unneeded, or even silly to you. But somebody, somewhere, is doing stream processing of TB-sized JSON objects, and they will care. These news are for them.

I remember when I was coming up on the command line and I'd browse the forums at unix.com. Someone would ask how to do a thing and CFAJohnson would come in with a far less readable solution that was more performative (probably replacing calls to external tools with Bash internals, but I didn't know enough then to speak intelligently about it now). People would say, "Why use this when it's harder to read and only save…

Why are half the forum posts on there all about AI? Yikes

Re: A Faster Alternative to Jq

#247

Post by sophacles »

Kovah wrote: post #5

I wonder so often about many new CLI tools whose primary selling point is their speed over other tools. Yet I personally have not encountered any case where a tool like jq feels incredibly slow, and I would feel the urge to find something else. What do people do all day that existing tools are no longer enough? Or is it that kind of "my new terminal opens 107ms faster now, and I don't notice it, but I simply feel bet…

I'll write a one-off shell pipeline to inspect something on 10^5 servers - it will be sent to each of those servers and run once or a handful of times, and the results will be transmitted back and that's that. Kind of a map-reduce shell thing, for ops type tasks.

Sometimes those will actually need to process through a bunch of data unexpectedly.

Sometimes those will be run on a loop - once per second, N per minute (etc), and the results will be used to monitor a situation until a bug is fixed or a spike in load is resolved or a proper monitoring program/metric can be deployed.

Sometimes those are to investigate a pegged CPU and the amortized lower runtime across all the tasks on the CPU is noticable.

We run our machines hot and part of the reason we can do that is being in the habit of choosing lower cost (in cycles) tooling whenever we can. If i can spend a little time and effort learning a tool that saves a bunch of cpu in aggregate, its a win. When the whole company does it, we can spend a lot less on hardware than it costs in engineer time to make these decisions.

Another way of putting it is: its a type of frugality (not cheapness, just spending wisely). If you save a dollar once, its nothing. If you have a habit of saving a dollar every time the opportunity arises, it adds up quickly. By having a habit of choosing more performant tools, you're less likely to hit a case where you wish you did use more performant tools, and are practiced at it when the need arises for pure parsimony and it's less painful.

Re: A Faster Alternative to Jq

#248

Post by mememememememo »

lock1 wrote: post #99
mememememememo wrote: post #93

Earlier quoted context omitted.

Also as someone who looks at latency charts too much, what happens is a request does a lot in series and any little ms you can knock off adds up. You save 10ms by saving 10 x 1ms. And if you are a proxyish service then you are a 10ms in a chain that might be taking 200 or 300ms. It is like saving money, you have to like cut lots of small expenses to make an impact. (unless you move etc. but once you done that it is s…

Wait what? I don't get why performance improvement implies reliability and incident improvement. For example, doing dangerous thing might be faster (no bound checks, weaker consistency guarantee, etc), but it clearly tend to be a reliability regression.

Somes directly like other commenters touch on. Less likely to saturate CPU quickly. Lower cost to run so can have more headroom.

But also the stuff you tend to do to make it fast makes it more reliable.

Local caches reduce network traffic. Memory is more reliable than network IO so it improves reliability.

Reducing lookup calls to other services (e.g. by supplying context earlier in the dependency chain) makes it faster and more reliable.

Your code will probably branch less and become more predictable too.

And often the code is simpler (sometimes not when a performance hack is used)

Re: A Faster Alternative to Jq

#250

Post by quotemstr »

maleldil wrote: post #187
quotemstr wrote: post #30

Reminder you can also get DuckDB to slurp the JSON natively and give you a much more expressive query model than anything jq-like.

How does it deal with nested objects? E.g. one of the fields/columns is an array of objects.

Beautifully. UNNEST works well, as do the pivot operators.
Post reply on HN