Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster and simpler with the command line: deep-comparing JSON files with jq

genius.engineering

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#61

Post by jeroenjanssens »

pjungwir wrote: post #55

I really enjoyed this article, and I think it shows how successfully jq fits into the Unix culture of sed/awk/grep/etc. It seems so rare to find new CLUI tools that feel as "classical" as jq. It has helped me do one-off tasks like this several times, but I've really only scratched the surface. Often with newer tools I'm reluctant to invest in going deeper into really learning the features, but with jq I have a lot of…

Chapter 5 of Data Science at the Command Line (O'Reilly, 2014) mentions `jq` briefly: https://www.datascienceatthecommandline.com/chapter-5-scrubb...

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#62

Post by Groxx »

Drdrdrq wrote: post #51
mattbillenstein wrote: post #42

Earlier quoted context omitted.

Yikes, that's nasty.

It is. But it's a problem of JSON itself, not just jq.

TIL: JSON has no specified number implementation: http://www.ecma-international.org/publications/files/ECMA-ST...

>JSON is agnostic about the semantics of numbers ... JSON instead offers only the representation of numbers that humans use: a sequence of digits.

So... anything is valid, per the spec.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#63

Post by hannasanarion »

GordonS wrote: post #57

> What’s the best way to compare these two 5GB files? A much simpler way to do this is simply to hash the files, for example using sha256sum, which AFAIK ships with just about every Linux distro. Then just compare the hashes.

Having the same content is not the same as being identical verbatim.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#64

Post by kemitche »

pritambaral wrote: post #58
iooi wrote: post #53

Earlier quoted context omitted.

JSON != JavaScript > echo 1152921504606846976 | python -c 'import sys, json; print(json.load(sys.stdin))' 1152921504606846976

Python's json package != JSON JSON: https://tools.ietf.org/html/rfc8259#page-8

The link says that it's up to the implementation, which means it's valid for Python's JSON implementation to support larger numbers.

It's less "interoperable" but not strictly invalid, by my read.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#65

Post by yonatank »

pritambaral wrote: post #58
iooi wrote: post #53

Earlier quoted context omitted.

JSON != JavaScript > echo 1152921504606846976 | python -c 'import sys, json; print(json.load(sys.stdin))' 1152921504606846976

Python's json package != JSON JSON: https://tools.ietf.org/html/rfc8259#page-8

It looks like JSON doesn't specifically define how numeric numbers should be stored. It just recommends expecting precision up to the double precision limits.

Still interesting to know it's not just a jq quirk.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#66

Post by rane »

gerbilly wrote: post #59

I wanted to like jq, but honestly, I can't figure out it's crazy syntax.

If you're into JavaScript (or LiveScript) or functional programming, you might find ramda-cli[1] more palatable. Disclaimer: I've created it.

[1]: https://github.com/raine/ramda-cli

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#67

Post by rzzzt »

iooi wrote: post #53
Drdrdrq wrote: post #51

Earlier quoted context omitted.

It is. But it's a problem of JSON itself, not just jq.

JSON != JavaScript > echo 1152921504606846976 | python -c 'import sys, json; print(json.load(sys.stdin))' 1152921504606846976

The problem is made worse on the receiving end (the browser). I've ran into this issue when serialization libraries in Java send a 64-bit long value as a sequence of digits, then things over ~50 bits get silently truncated, you find out about it, then switch to quoted strings.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#69

Post by [deleted] »

scarejunba wrote: post #48
orf wrote: post #40

Earlier quoted context omitted.

Yes, you're totally correct. Using a heavyweight solution like JSON is beyond the pale, I should use a much more lightweight approach involving a database server. Your tone is oddly superior in your reply, which is really at odds with the technical content of your messages. > if record fields are consistent This is all very confused. The issue is that the JSON fields where not consistent compared to the baseline. So…

Like a SQLite DB? Actually, why don't we just transfer stuff as SQLite DBs. Single file, built-in schema, you can index. I mean, HDF is super-general and stuff, but it looks like SQLite would solve all the trouble with CSVs.

[deleted]

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#70

Post by sigi45 »

pjungwir wrote: post #55

I really enjoyed this article, and I think it shows how successfully jq fits into the Unix culture of sed/awk/grep/etc. It seems so rare to find new CLUI tools that feel as "classical" as jq. It has helped me do one-off tasks like this several times, but I've really only scratched the surface. Often with newer tools I'm reluctant to invest in going deeper into really learning the features, but with jq I have a lot of…

I had the opposite thought when I used it for the first time.

Even when reading the article I thought about it :)

Post reply on HN