Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster and simpler with the command line: deep-comparing JSON files with jq

genius.engineering

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#81

Post by kibibu »

aditya wrote: post #80
yonatank wrote: post #39

Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000

Not jq but JS itself. Here's the epic jwz rant on it: https://www.jwz.org/blog/2010/10/every-day-i-learn-something... with Brendan Eich chiming in: https://www.jwz.org/blog/2010/10/every-day-i-learn-something...

Twitter had to switch their tweet id representation in the API to handle this - it was numeric and switched to strings.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#82

Post by saagarjha »

pathseeker wrote: post #78
saagarjha wrote: post #77

Earlier quoted context omitted.

awk, sed?

For extracting a value out of json where the keys can arbitrarily re-order and you have nested maps containing the same key names? No thanks.

I was providing them as examples of things "perfect for writing quick shell one liners", not good JSON parsers.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#84

Post by rajangdavis »

paulddraper wrote: post #30
jb3689 wrote: post #2

It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I still don't think Ruby is the best choice for doing raw JSON parsing. Last time I had to care about JSON speed we were transforming billions of events and the Ruby JSON lib was becoming a bottleneck

> It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I am on the edge of my seat now. Would you mind listing which libraries are much (say, an order of magnitude) faster?

Don't think it's an order of magnitude faster, but oj is supposed to be the standard for Ruby.

https://github.com/ohler55/oj

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#85

Post by int0x80 »

otterley wrote: post #43
yonatank wrote: post #39

Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000

This is an artifact of JavaScript, which even as of ES6 uses IEEE 754 double-precision floats for all numeric values. jq likely uses the same implementation internally for compatibility reasons and to avoid surprises of a different kind. See https://www.ecma-international.org/ecma-262/6.0/#sec-ecmascr...

I think an 'error out if overflow/truncation'-mode available as a command line flag could be useful if you just don't have any JS involved in the JSON pipeline.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#86

Post by chokma »

aditya wrote: post #80
yonatank wrote: post #39

Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000

Not jq but JS itself. Here's the epic jwz rant on it: https://www.jwz.org/blog/2010/10/every-day-i-learn-something... with Brendan Eich chiming in: https://www.jwz.org/blog/2010/10/every-day-i-learn-something...

Note: the links currently redirect to https://imgur.com/32R3qLv (image of a testicle and derogatory comment on HN)

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#87

Post by gerbilly »

saagarjha wrote: post #77
Rotten194 wrote: post #71

Earlier quoted context omitted.

it's a bit weird, but it's perfect for writing quick shell one liners with once you get used to it. no regular scripting language can match it for that

awk, sed?

The braces make that more difficult, awx and sed can work with YAML more easily, but awk works best on column oriented data, and YAML and JSON are more row oriented.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#88

Post by JulianMorrison »

cryptonector wrote: post #24
JulianMorrison wrote: post #12

Earlier quoted context omitted.

jq is a C program. In theory a truly specific program could work better. In practise, the broad scope of jq allows you to discover the operations you need and respond to changes in requirements without being locked into custom code, and any given programmer probably couldn't do the same job better.

jq is a C program, yes, but jq programs are interpreted. Because jq is a dynamically-typed language, it wouldn't be easy to compile it to object code that would run too much faster than the byte-interpreted version (though it would still run faster). As you say, jq's power is that it is an expressive language, and it's much much easier to write jq programs that work than it is to write C/C++ programs as needed that d…

The interpreted language is just the setup phase for a pipeline of compiled-in data transformations.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#89

Post by Filligree »

dith3r wrote: post #34
theamk wrote: post #21

Earlier quoted context omitted.

(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby or (2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ). Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once. The streaming API might work better with jq-based prepro…

Python is fast at parsing JSON, Go had hard time to match parsing speed of it. Additionally you have PyPy to help.

Python is fast at doing anything that doesn't involve running Python.

That's an important caveat. Python's C JSON parser library is super-fast, but if you want to use the data for anything but a simple equality check afterwards, it'll be slow as molasses.

Or you'll write a C extension for it...

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#90

Post by baud147258 »

chokma wrote: post #86
aditya wrote: post #80

Earlier quoted context omitted.

Not jq but JS itself. Here's the epic jwz rant on it: https://www.jwz.org/blog/2010/10/every-day-i-learn-something... with Brendan Eich chiming in: https://www.jwz.org/blog/2010/10/every-day-i-learn-something...

Note: the links currently redirect to https://imgur.com/32R3qLv (image of a testicle and derogatory comment on HN)

Copy-pasting the link bypass this. It's using the HN referrer (I think?) to redirect to imgur
Post reply on HN