Surprised to see that there's no official binaries for arm64 darwin. Meaning macOS users will have to run it through the Rosetta 2 translation layer.
A Faster Alternative to Jq
21–30 of 281 posts
Re: A Faster Alternative to Jq
#22From their README [0]:
> Jq is a powerful tool, but its imperative filter syntax can be verbose for common path-matching tasks. jsongrep is declarative: you describe the shape of the paths you want, and the engine finds them.
IMO, this isn't a common use case. The comparison here is essentially like Java vs Python. Jq is perfectly fine for quick peeking. If you actually need better performance, there are always faster ways to parse JSON than using a CLI.
Re: A Faster Alternative to Jq
#23Earlier quoted context omitted.
> Now I'm really curious. What field are you in that ndjson files of that size are common? I'm not OP,but structured JSON logs can easily result in humongous ndjson files, even with a modest fleet of servers over a not-very-long period of time.
So what's the use case for keeping them in that format rather than something more easily indexed and queryable? I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.
The use case could be e.g. exactly processing an old trove of logs into something more easily indexed and queryable, and you might want to use jq as part of that processing pipeline
Re: A Faster Alternative to Jq
#24If it's easier to use than jq, they should sell the tool on that.
Re: A Faster Alternative to Jq
#25Surprised to see that there's no official binaries for arm64 darwin. Meaning macOS users will have to run it through the Rosetta 2 translation layer.
I’d install it via cargo anyway and that would build it for arm64.
If the arm64 version was on homebrew (didn’t check if it is but assume not because it’s not mentioned on the page), I’d install it from there rather than from cargo.
I don’t really manually install binaries from GitHub, but it’s nice that the author provides binaries for several platforms for people that do like to install it that way.
Re: A Faster Alternative to Jq
#26Surprised to see that there's no official binaries for arm64 darwin. Meaning macOS users will have to run it through the Rosetta 2 translation layer.
Really? That is your response? This is an high quality article from someone who spend a lot of time implementing a cool tool and also sharing the intricate inner workings of it. And your response is, "eh there are no official binaries for my platform". Give them some credit! Be a little more constructive!
Re: A Faster Alternative to Jq
#27Earlier quoted context omitted.
So what's the use case for keeping them in that format rather than something more easily indexed and queryable? I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.
The use case could be e.g. exactly processing an old trove of logs into something more easily indexed and queryable, and you might want to use jq as part of that processing pipeline
Fair, but for a once-off thing performance isn't usually a major factor.
The comment I was replying to implied this was something more regular.
EDIT: why is this being downvoted? I didn't think I was rude. The person I responded to made a good point, I was just clarifying that it wasn't quite the situation I was asking about.
Re: A Faster Alternative to Jq
#28I learned a number of data processing cli tools: jq, mlr, htmlq, xsv, yq, etc; to name a few. Not to the level of completing advent of code or anything, but good enough for my day to day usage. It was never ending with the amount of formats I needed to extract data from, and the different syntax's. All that changed when I found nushell though, its replaced all of these tools for me. One syntax for everything, breath…
Same here, nushell is awesome! It helped me to automate so many more things than I did with any other shell. The syntax is so much more intuitive and coherent, which really helps a lot for someone who always forgot how to write ifs or loops in bash ^^
Re: A Faster Alternative to Jq
#29Earlier quoted context omitted.
> Now I'm really curious. What field are you in that ndjson files of that size are common? I'm not OP,but structured JSON logs can easily result in humongous ndjson files, even with a modest fleet of servers over a not-very-long period of time.
So what's the use case for keeping them in that format rather than something more easily indexed and queryable? I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.
Replying here because the other comment is too deeply nested to reply.
Even if it's once off, some people handle a lot of once-offs, that's exactly where you need good CLI tooling to support it.
Sure jq isn't exactly super slow, but I also have avoided it in pipelines where I just need faster throughput.
rg was insanely useful in a project I once got where they had about 5GB of source files, a lot of them auto-generated. And you needed to find stuff in there. People were using Notepad++ and waiting minutes for a query to find something in the haystack. rg returned results in seconds.
Re: A Faster Alternative to Jq
#30Reminder you can also get DuckDB to slurp the JSON natively and give you a much more expressive query model than anything jq-like.