Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Faster Alternative to Jq

micahkepe.com

Post reply on HN
Print view View original

221–230 of 281 posts

Re: A Faster Alternative to Jq

#221

Post by 1vuio0pswjnm7 »

One problem I have not seen addressed by jq or alterataives, perhaps this one addresses it, is "JSON-like" data. That is, JSON that is not contained in a JSON file

For example, web pages sometimes contain inline "JSON". But as this is not a proper JSON file, jq-style utilties cannot process it

The solution I have used for years is a simple utility written in C using flex^1 (a "filter") that reformats "JSON" on stdin, regardless of whether the input is a proper JSON file or not, into stdout that is line-delimited, human-readable and therefore easy to process with common UNIX utilities

The size of the JSON input does not affect the filter's memory usage. Generally, a large JSON file is processed at the same speed with the same resource usage as a small one

The author here has provided musl static-pie binaries instead of glibc. HN commenters seeking to discredit musl often claim glibc is faster

Personally I choose musl for control not speed

1. jq also uses flex

Re: A Faster Alternative to Jq

#224

Post by dhuan_ »

regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

I agree, even trivial tasks require us to go back to jq's manual to learn how to write their language.

this and other reasons is why I built: https://github.com/dhuan/dop

Re: A Faster Alternative to Jq

#225

Post by NoSalt »

DoctorOW wrote: post #203
NoSalt wrote: post #144

Earlier quoted context omitted.

> "somebody, somewhere, is doing stream processing of TB-sized JSON objects" That's crazy to think about. My JSON files can be measured in bytes. :-D

All files can be measured in bytes. :)

You, sir or ma'am, are a first class smarty pants.

Re: A Faster Alternative to Jq

#226

Post by raydev »

regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

Like I did with regex some years earlier, I worked on a project for a few weeks that required constant interactions with jq, and through that I managed to lock in the general shape of queries so that my google hints became much faster.

Of course, this doesn't matter now, I just ask an LLM to make the query for me if it's so complex that I can't do it by hand within seconds.

Re: A Faster Alternative to Jq

#227

Post by randusername »

nh23423fefe wrote: post #220
randusername wrote: post #186

Earlier quoted context omitted.

I'm not GP, I use jq all the time, but I each time I use it I feel like I'm still a beginner because I don't get where I want to go on the first several attempts. Great tool, but IMO it is more intuitive to JSON people that want a CLI tool than CLI people that want a JSON tool. In other words, I have my own preconceptions about how piping should work on the whole thing, not iterating, and it always trips me up. Here'…

oh my god how could I have been doing this for so long and not realize that you can redirect before your binary.

I knew cat was an anti-pattern, but I always thought it was so unreadable to redirect at the end

Re: A Faster Alternative to Jq

#228

Post by allknowingfrog »

I deal with a fair amount of newline-delimited JSON in my day job, where each line in the file is a complete JSON object. I've seen this referred to as "jsonl", and it's not entirely uncommon for logs and other kinds of time-series data dumps. Do any of the popular JSON CLI tools work with this format? I didn't see any mention of it here.

Re: A Faster Alternative to Jq

#229

Post by chuckadams »

figmert wrote: post #209
randusername wrote: post #186

Earlier quoted context omitted.

I'm not GP, I use jq all the time, but I each time I use it I feel like I'm still a beginner because I don't get where I want to go on the first several attempts. Great tool, but IMO it is more intuitive to JSON people that want a CLI tool than CLI people that want a JSON tool. In other words, I have my own preconceptions about how piping should work on the whole thing, not iterating, and it always trips me up. Here'…

Here's an easier to understand query for what you're trying to do (at least it's easier to understand for me): cat input.json | jq -r '(first | keys) as $cols | $cols, (.[] | [.[$cols[]]]) | @tsv' That whole map and from entries throws it off. It's not a good use for what you're doing. tsv expects a bunch of arrays, whereas you're getting a bunch of objects (with the header also being one) and then converting them to…

Honestly both of those make me do the confused-dog-head-tilt thing. I'd go for something sexp based, perhaps with infix composition, map, and flatmap operators as sugar.

Re: A Faster Alternative to Jq

#230

Post by LgWoodenBadger »

regus wrote: post #137

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

I completely agree. I much prefer leveraging actual javascript to get what I need instead of spending time trying to fumble my way through jq syntax.
Post reply on HN