Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zq: An easier and faster alternative to jq

brimdata.io

Post reply on HN
Print view View original

201–210 of 237 posts

Re: Zq: An easier and faster alternative to jq

#201

Post by jodrellblank »

mschuster91 wrote: post #131
jodrellblank wrote: post #118

Earlier quoted context omitted.

The GDPR applies to personal data. PowerShell telemetry isn't personal data, so it's not covered by the GDPR. What is reported is documented here: https://docs.microsoft.com/en-us/powershell/module/microsoft... and is " anonymized information about the host running PowerShell, and information about how PowerShell is used ". It sucks that it has telemetry, but anonymised information about whether a computer ran 10 .ex…

Even the fact that a particular piece of software is used by a specific IP address is enough PII that it's covered under GDPR by most viewpoints. The fact that Microsoft is collecting even more data doesn't excuse telemetry in PowerShell at all . I would simply wish for no telemetry to happen at all without user consent. If Microsoft wants information about how people use their software or how stable it is and not en…

> "Even the fact that a particular piece of software is used by a specific IP address is enough PII that it's covered under GDPR by most viewpoints."

I draw your attention to the link I posted, and the purple background call out box with the exclamation mark icon and the heading "Note" which says: "Application Insights uses the hosts IP address to determine the geographic location. The IP address is never included in the telemetry data or stored in the database."

> "I would simply wish for no telemetry to happen at all without user consent."

I would, too. So did someone on Github: https://github.com/PowerShell/PowerShell/issues/15722 "Change telemetry from opt-out to opt-in" where Microsoft said "we felt that by making telemetry opt-in it would bias and limit our telemetry in a way that would make less useful to our users."

Re: Zq: An easier and faster alternative to jq

#202

Post by spiralx »

bblb wrote: post #72
brushfoot wrote: post #5

The name of its corporate progenitor may leave a bad taste in some mouths, but I highly recommend PowerShell for this sort of thing. It's cross platform, MIT licensed, and comes with excellent JSON parsing and querying capabilities. Reading, parsing, and querying JSON to return all red cars: Get-Content cars.json | ConvertFrom-Json | ? { $_.color -eq 'red' } The beauty of this is that the query syntax applies not jus…

PowerShell is "Python interactive done right". It's too bad it has a bad rap in open source community and it might never get the traction it really deserves. Sure it has it's downsides, which tech doesn't, but PowerShell has solved so many issues and annoyances with the shells that we've been used to, that it still comes out as the winner. I've been using it since day one from 2006, every single day. It has come a lo…

> PowerShell is "Python interactive done right".

Actually PowerShell is "Perl interactive done right" if you read what the designers say about their influences - the automatic variable $_ is straight from Perl and the array creation syntax @(a, b, c) is also a Perl-ism from @arr = (a, b, c). Which is funny as I dislike Perl intensely but really like PowerShell :)

To be fair there's not much Perl in PS, it's as much influenced by KSH, Awk, cmd.exe and VBScript as Perl. Thankfully "influenced by" isn't "a melange of", because a combination of all of those sounds like an abomination lol, and PS is wonderful in being about as consistent and simple as a proper shell can get.

Re: Zq: An easier and faster alternative to jq

#203

Post by zeroimpl »

Beltalowda wrote: post #57
psacawa wrote: post #6

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lackin…

> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forg…

I use both awk and jq infrequently enough that I tend to struggle with anything non-trivial. I think zq would be the same.

> Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forgetting it.

I think this is the main thing. I’d prefer a streamlined CLI tool where you passed in some JS code and it’d just run it on the input (with the same slurp/raw args as jq). Could just be npm with underscore.js autoimported.

Re: Zq: An easier and faster alternative to jq

#204

Post by phibz »

I think of

echo '1 2 3' | jq ....

as creating three separate json documents, each with a single number as their top level "document" , body, or content.

So of course you can't sum them. They are fed as separate documents to the jq pipeline as if you processed three separate jq commands.

Perhaps by stateless you mean no mutuable global state? But it certainly maintains state from the location in the input document to the output of each selector/functor.

IMO it helps if you have a background in some of the concepts of functional programming.

Re: Zq: An easier and faster alternative to jq

#205

Post by anitil »

laurent123456 wrote: post #103
Beltalowda wrote: post #57

Earlier quoted context omitted.

> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forg…

I wonder if someone tried to use plain JS as a filtering language? It would be more verbose but it would be easy to remember. For example: [1,2,3] | js "out = 0; for (const n of this) out += n" That would print "6". `out` would be a special variable you write to to print the result, and `this` would be the input.

That's a really interesting suggestion, similar to how AWK uses $0, $1 etc.

Re: Zq: An easier and faster alternative to jq

#206

Post by mathfailure »

lillecarl wrote: post #153
kalev wrote: post #101

Earlier quoted context omitted.

Please check FZF [1] and it’s integration with ctrl-r. It’s a huge productivity boost and I cannot live without it. [1] https://github.com/junegunn/fzf

There's also McFly[1] that does interactive history search. [1]: https://github.com/cantino/mcfly

This sounds quite awesome, it's a pity that the realization is quite 'alien' to unix way of things. IMO shell commands history should be a database by default, not just a single file that gets auto-appended with both awesome commands and crap you don't really want to get stored for later.

Thanks for linking this project, will try it, it may be a game changer.

Re: Zq: An easier and faster alternative to jq

#207

Post by anitil »

eloh wrote: post #191
sfink wrote: post #178

The thing that I find myself wanting, which is lacking in both jq and zq afaik, is interactive exploration. I want to move around in a large JSON file, narrow my context to the portion I'm interested in, and do specialized queries and transformations on just the data I care about. I wrote a tool to do this -- https://github.com/hotsphink/sfink-tools/blob/master/bin/jso... -- but I do not recommend it to anyone other…

You could take a look at jless [1], it allows interactive selections/browsing in JSON documents. [1] https://jless.io/

Do you know a way to copy the current selection in jless to the paste buffer? I find myself narrowing down a query for jq using jless but then having to manually remember the query to then jump to my terminal.

Re: Zq: An easier and faster alternative to jq

#208

Post by anitil »

AcerbicZero wrote: post #71

I'm pretty new to jq (maybe 2 years of exposure) but from my perspective - on some level, jq does to json what powershell does to everything windows, except powershell gives me the get-member cmdlet, so when I don't know what is even in my object, I can explore. Sometimes jq -r '.[]' works, but its all just trial and error. I use plenty of jq in my scripts, but I can never seem to visualize how jq looks at the data.…

I use jless to manually find what I'm looking for and then using the result as a starting point. Unfortunately I don't know how to get that query in to the paste buffer yet so there's a manual step in the middle

Re: Zq: An easier and faster alternative to jq

#209

Post by spiralx »

xg15 wrote: post #167

So, admitted jq fanboy here, but I found a lot of the criticism from the articale really sensible. I think jq has a pretty elegant data model, but the syntax is often very clunky to work with. So here is a half thought-out idea how you might improve the syntax for the "stateful operations" usecase the OP outlined: I think it's not quite true that different elements of a sequence can never interact. The OP mentioned r…

My thought is that having | as the pipe operator in the shell means that it's okay to use | for the or operator in your query syntax as the people using your XGQ will be familiar with that usage, but to then introduce a pipe operator (which >> pretty much is) alongside using | as something that isn't a pipe is just going to fuck with your muscle memory for reading code. Especially as >> is already the right-shift operator, and so you expect the precedence for 'a | b >> c' to mean 'a | c(b)'.

The pipe operator that's in its final stages of approval for JavaScript uses '|>' as its sigil, which is a decent compromise between not conflicting with existing operators, being compatible with developers' existing pattern matching, and representing what it does somewhat. And 'a | b |> c' is ok.

You could just have '@c a | b' mean "everything from @c onwards to the end of the string is the argument to c" i.e. 'c(a | b)' and have 'c a | b' be 'c(a) | b', then anything more complicated just requires using the parentheses operator to enclose an expression i.e. 'c (a | b)' or just 'c(a | b)' if your tokenizer is a bit cleverer :) Actually I like this idea, because '@' is syntactic sugar for () around the rest of the query, and a function then operates on the value of the expression following it.

Re: Zq: An easier and faster alternative to jq

#210

Post by anitil »

arwineap wrote: post #8

I've never found jq to be particularly hard, or slow

I've generally found it a bit hard to get an initial query going, but then it is as fast as anything I've thrown at it. Generally something else (probably my code) falls over before I even notice jq's impact.
Post reply on HN