Live data from Hacker News

Zq: An easier and faster alternative to jq

brimdata.io

191–200 of 237 posts

Re: Zq: An easier and faster alternative to jq

#191
post #178

The thing that I find myself wanting, which is lacking in both jq and zq afaik, is interactive exploration. I want to move around in a large JSON file, narrow my context to the portion I'm interested in, and do specialized queries and transformations on just the data I care about. I wrote a tool to do this -- https://github.com/hotsphink/sfink-tools/blob/master/bin/jso... -- but I do not recommend it to anyone other…

You could take a look at jless [1], it allows interactive selections/browsing in JSON documents.

[1] https://jless.io/

Re: Zq: An easier and faster alternative to jq

#192
post #178

The thing that I find myself wanting, which is lacking in both jq and zq afaik, is interactive exploration. I want to move around in a large JSON file, narrow my context to the portion I'm interested in, and do specialized queries and transformations on just the data I care about. I wrote a tool to do this -- https://github.com/hotsphink/sfink-tools/blob/master/bin/jso... -- but I do not recommend it to anyone other…

[deleted]

Re: Zq: An easier and faster alternative to jq

#193

I see a lot of JQ experts on this thread, so I'll bite the bullet here as a novice. The purpose of life is not to know JQ. I just want to process the JSON so I can move on and do whatever is actually important. Ideally, I'd just be able to tell GPT-codex to do what I want to do to the JSON in English. We're not there yet, but in the meantime if there's another tool that allows me to know less in exchange for doing mo…

[deleted]

Re: Zq: An easier and faster alternative to jq

#194
post #178

The thing that I find myself wanting, which is lacking in both jq and zq afaik, is interactive exploration. I want to move around in a large JSON file, narrow my context to the portion I'm interested in, and do specialized queries and transformations on just the data I care about. I wrote a tool to do this -- https://github.com/hotsphink/sfink-tools/blob/master/bin/jso... -- but I do not recommend it to anyone other…

One solution I’ve seen is basically to hijack fzf to interactively input a jq query, add closing brackets in a naive way, run jq -C … | head on an input file, and display the result as a fzf ‘preview’. fzf ends up handling things like the preview command and display and line-editing logic but it may be slow if you don’t get early results.

Re: Zq: An easier and faster alternative to jq

#195
post #174

Earlier quoted context omitted.

> However, presumably PowerShell requires at least the .NET Runtime if not the .NET SDK, doesn't it? Nope, these days .NET programs (like PowerShell) bundle the runtime. But even if they did a lighter distribution that depended on the runtime already being installed, there would be no .NET telemetry sent. > Does running the recommended "dotnet --list-runtimes" command send telemetry, like most of the commands? This i…

So the "dotnet" command is only in the SDK, not in the separately downloadable Runtime? Does the Runtime have some other command to launch an executable? Edit: Actually, the ".NET Runtime 6.0.4" [1] (not the SDK) definitely has a "dotnet" command included. Presumably with the telemetry? [1] https://dotnet.microsoft.com/en-us/download/dotnet/6.0

When I say "the runtime", I'm referring to everything that would be bundled into a published .NET program. The base class libraries, the bootstrapper, etc. There is no telemetry here.

Yes, if you download a .NET Runtime distribution, it will include the `dotnet` command from the SDK so that basic commands like `dotnet --list-runtimes` and `dotnet --list-sdks` are available. These commands may send telemetry. But as you probably saw on https://docs.microsoft.com/en-us/dotnet/core/tools/telemetry , using `dotnet path/to/program.dll` to run an unbundled .NET program will never send telemetry.

Re: Zq: An easier and faster alternative to jq

#196
post #124

The name of its corporate progenitor may leave a bad taste in some mouths, but I highly recommend PowerShell for this sort of thing. It's cross platform, MIT licensed, and comes with excellent JSON parsing and querying capabilities. Reading, parsing, and querying JSON to return all red cars: Get-Content cars.json | ConvertFrom-Json | ? { $_.color -eq 'red' } The beauty of this is that the query syntax applies not jus…

Powershell's object pipes are more inspectable than any of Bourne's text-based decendants. But the tool itself occupies a niche between "write shell, dealing with esoteria of grep/sed/awk/jq/etc" and "write python getting constructs than handle complexity better than pipes". Looking at the popularity of VSCode, I don't think Microsoft hatred blocks its adoption.

> Looking at the popularity of VSCode, I don't think Microsoft hatred blocks its adoption.

In-apt comparison. The people using VS Code are more likely to be migrating from proprietary tools like PyCharm, Sublime Text, etc or bloated offerings like NetBeans or roughly equivalent offerings like Atom.

The people that would use PowerShell would be migrating from the likes of Zsh, Bash, Fish, and other “hard core free” software.

Re: Zq: An easier and faster alternative to jq

#198

Earlier quoted context omitted.

> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forg…

I wonder if someone tried to use plain JS as a filtering language? It would be more verbose but it would be easy to remember. For example: [1,2,3] | js "out = 0; for (const n of this) out += n" That would print "6". `out` would be a special variable you write to to print the result, and `this` would be the input.

Not quite that, but ramda-cli[1] which I've created solves this problem, at least for me, by offering the familiar set of functions from Ramda, and you can create pipelines with those to do operations on your data.

[1]: https://github.com/raine/ramda-cli

Re: Zq: An easier and faster alternative to jq

#199

Earlier quoted context omitted.

I don't know; I wouldn't presume to tell you what you do or don't find arcane, but once I understood the somewhat unusual flow of awk ("for every line, check if the line matches this condition, and if it does run this block of code") I found it's quite easy to work with. It's "arcane" in the sense that it has an implicit loop and that it's a specialized language for a very limited class of problems, but I found that…

> an implicit loop As an occasional awk user, I'd love if you expand on this. Maybe it will help clear things up for me. You're not referring to the fact that awk operates on every line independently, are you?

To expand on the other reply, there are a couple more implicit loops. There's a loop over all of the command line arguments/files, then a loop for every line in each of those files, then there is kind of a loop over the whitespace delimited fields of each of those lines. The main thing that helped me understand AWK was that every block in a script is just a pattern/action pair. When I saw snippets like

  ... | awk '{print $2}' 
I thought there was all this confusing syntax, but something like

  awk '/pattern/ {print}'
was more clear to me. In the first case, the empty pattern matches every line of the input, and the action is simply to print the second field of each line. Patterns can vary in complexity from the empty pattern to long chains of logical operators and regular expressions, such as /pattern/ in the second example. The outer quotes are just to prevent the shell from eating your dollar signs or other special characters. In a standalone AWK script you can write it like

  /pattern/ {
    print
  }
which also makes it look more like another language.

If you can get your hands on a copy of The AWK Programming Language, it's a pretty quick and pleasant read that helped everything make more sense to me. I do most of my data analysis for my research using AWK and really enjoy working with it.

Re: Zq: An easier and faster alternative to jq

#200
Looks interesting, and I like the analysis of JQ's model in order to compare and contrast it with ZQ. It reminds me somewhat of the difference between XPath 1 to XPath 2 - the former worked great if you were selecting stuff and using the built-in functions to do some simple transformations but if you tried to go further it was a hot mess of incompatible types each with their own constraints and quirks. In XPath 2 they completely revised the data model and type system, making everything an atomic value, node or a sequence of values and/or nodes, allowing types to be queried and operated on, and added a ton of useful built-in functions and operators, making impossible queries in XPath 1 possible.

https://docs.microsoft.com/en-us/archive/msdn-magazine/2003/...

Anyway, I've installed ZQ and will look to use it, even my simple usage of JQ had already led to thoughts of writing my own, better version :)

Quick bug report: On the Aggregate Functions page the link to _countdistinct_ goes to the page for _count_, and there actually isn't a page at https://zed.brimdata.io/docs/language/aggregates/countdistin....

Post reply on HN