I'm glad you adjusted the CSS while I was typing my comment. I needed to switch to dark mode to be able to read highlighted words.
Nice write up. I will try out your tool.
A Faster Alternative to Jq
171–180 of 281 posts
Re: A Faster Alternative to Jq
#172Having used `jq` and `yq` (which followed from the former, in spirit), I have never had to complain about performance of the _latter_ which an order of magnitude (or several) _slower_ than the former. So if there's something faster than `jq`, it's laudable that the author of the faster tool accomplished such a goal, but in the broader context I'd say the performance benefit would be required by a niche slice of the u…
I use jq to grind through gigabytes of GeoJSON files exported from ArcGIS, as an ETL stage. It takes a long time.
Re: A Faster Alternative to Jq
#173Earlier quoted context omitted.
You really need to go and learn about the concept of determinism and why for some tasks we need and want deterministic solutions. It's an important idea in computer science. Go and learn.
You need to learn to adapt to the real world where most things are not deterministic. Go and learn.
That may be true, but do you not want determinism where possible, especially within this context, i.e. filtering data?
Re: A Faster Alternative to Jq
#174Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.
LOL ... I can absolutely feel your pain. That's exactly why I created for myself a graphical approach. I shared the first version with friends and it turned into "ColumnLens" (ImGUI on Mac) app. Here is a use case from the healthcare industry: https://columnlens.com/industries/medical
Re: A Faster Alternative to Jq
#175Re: A Faster Alternative to Jq
#176Earlier quoted context omitted.
Wait what? I don't get why performance improvement implies reliability and incident improvement. For example, doing dangerous thing might be faster (no bound checks, weaker consistency guarantee, etc), but it clearly tend to be a reliability regression.
First, if a performance optimization is a reliability regression, it was done wrong. A bounds check is removed because something somewhere else is supposed to already guaratee it won't be violated, not just in a vacuum. If the guarantee stands, removing the extra check makes your program faster and there is no reliability regression whatsoever. And how does performance improve reliability? Well, a more performant ser…
"Removing an extra check", so there is a check, so the check is not removed?
Re: A Faster Alternative to Jq
#177Earlier quoted context omitted.
You really need to go and learn about the concept of determinism and why for some tasks we need and want deterministic solutions. It's an important idea in computer science. Go and learn.
You need to learn to adapt to the real world where most things are not deterministic. Go and learn.
Is your argument that the world isn't deterministic and so we should also apply nondeterminism to filtering json data?
Re: A Faster Alternative to Jq
#178Earlier quoted context omitted.
This isn't for you then > The query language is deliberately less expressive than jq's. jsongrep is a search tool, not a transformation tool-- it finds values but doesn't compute new ones. There are no filters, no arithmetic, no string interpolation. Mind me asking what sorts of TB json files you work with? Seems excessively immense.
> Uses jq for TB json files > Hadoop: bro > Spark: bro > hive: bro > data team: bro
made me remember this article
https://adamdrake.com/command-line-tools-can-be-235x-faster-...>
Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
Conclusion: Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.Re: A Faster Alternative to Jq
#179Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.
I just ask Opus to generate the queries for me these days.
Re: A Faster Alternative to Jq
#180I'm glad you adjusted the CSS while I was typing my comment. I needed to switch to dark mode to be able to read highlighted words. Nice write up. I will try out your tool.
LOL ... came here to grips about that!
Also "jg" reads very similar to "jq", and initially I thought he was talking about "jq" all along, and I was like: where can I see the "jasongrep" examples? Threw me off for a minute.