Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster and simpler with the command line: deep-comparing JSON files with jq

genius.engineering

Post reply on HN
Print view View original

31–40 of 93 posts

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#31

Post by wwarnerandrew »

totally wrote: post #20

That's cool, you made a thing that verifies the two export jobs you wrote have the same data even though they have different output. I can't help wondering, if you control the code that generates the JSON, why not output in a conservative, consistent format? I'm sure there are pros/cons, but this work would allow something like `diff` to work, and then you don't have to maintain a separate utility.

good question! the analysis that I was doing was really a one-off for switching between these processes. We have unit tests and sanity checks to ensure consistency going forward, but as a final check before flipping the switch we wanted to be as confident as possible that we hadn't introduced any regressions across the full data-set.

The new export process is much more reliable and a _lot_ faster, but as a side effect of doing things in a different way it generated the export file in a different format. Given that the order of objects in an export file and the order of keys/etc in the JSON objects didn't matter for anything except comparing the two processes, I figured it was simpler to put the normalization logic in the one-off tool vs baking it into our export process. But certainly if we were maintaining both exports in an ongoing fashion and validating them against each other, it would make a lot more sense to spend time making sure they generated objects and keys in the same order.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#32

Post by wgjordan »

> My first thought was to write a ruby script to parse and compare the two exports, but after spending a little time coding something up I had a program that was starting to get fairly complicated, didn't work correctly, and was too slow—my first cut took well over an hour. Then I thought: is this one of those situations where a simple series of shell commands can replace a complex purpose-built script?

Key takeaway: next time, start with the second thought first and save yourself well over an hour!

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#34

Post by dith3r »

theamk wrote: post #21
baldeagle wrote: post #3

Earlier quoted context omitted.

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby or (2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ). Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once. The streaming API might work better with jq-based prepro…

Python is fast at parsing JSON, Go had hard time to match parsing speed of it. Additionally you have PyPy to help.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#36

Post by johnjohnsmith »

orf wrote: post #18
johnjohnsmith wrote: post #16

My problem with this article is the entire strategy for delivering data. A JSON file? That's probably the worst way I could think of: a CSV file would have been better. Part of the reason why databases exist is to handle exactly the problem the author is posing. There are already tools in place in SQL databases that track diffs for TB db's, and the authors could simply export patch files which would be far easier to…

CSV is hardly an easy format to parse, or really produce. There is no CSV standard, and I bet lyrics contain all kinds of weird characters that makes choosing a separator hard. Newline JSON is a fine interchange format for this, and the only advantage I can see for CSV is you can load it into a database in one command. Which begs the question as to why use a database at all for a simple one-off diff, when there are m…

Oh, I see: you probably didn't know that CSV formats is also means character separated values, and can actually use non-printing ASCII characters as delimiters. You didn't think I actually meant commas did you? I guess your experience with character separated value files is very limited. But my point was to illustrate if record fields are consistent you don't need a heavyweight solution like JSON... and you clearly missed my point.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#37

Post by cphoover »

I'm not sure why you are comparing the data to the old export instead of against a source of truth... for example what is in the upstream data source. Also why not verify using unit tests? Who is to say that the original export is valid and not the second export.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#38

Post by [deleted] »

baldeagle wrote: post #3
jb3689 wrote: post #2

It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I still don't think Ruby is the best choice for doing raw JSON parsing. Last time I had to care about JSON speed we were transforming billions of events and the Ruby JSON lib was becoming a bottleneck

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

[deleted]

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#39

Post by yonatank »

Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example:

  echo 1152921504606846976 | jq                                                            
  1152921504606847000

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#40

Post by orf »

johnjohnsmith wrote: post #36
orf wrote: post #18

Earlier quoted context omitted.

CSV is hardly an easy format to parse, or really produce. There is no CSV standard, and I bet lyrics contain all kinds of weird characters that makes choosing a separator hard. Newline JSON is a fine interchange format for this, and the only advantage I can see for CSV is you can load it into a database in one command. Which begs the question as to why use a database at all for a simple one-off diff, when there are m…

Oh, I see: you probably didn't know that CSV formats is also means character separated values, and can actually use non-printing ASCII characters as delimiters. You didn't think I actually meant commas did you? I guess your experience with character separated value files is very limited. But my point was to illustrate if record fields are consistent you don't need a heavyweight solution like JSON... and you clearly m…

Yes, you're totally correct. Using a heavyweight solution like JSON is beyond the pale, I should use a much more lightweight approach involving a database server.

Your tone is oddly superior in your reply, which is really at odds with the technical content of your messages.

> if record fields are consistent

This is all very confused. The issue is that the JSON fields where not consistent compared to the baseline. So now what? You suggest instead of investing time making them consistent, you should just switch format entirely and then make them consistent? Or are you suggesting that somehow a line of CSV is easier to compare than a line of JSON? Or I should now shove a bunch of non-printing ascii characters in my message and that's now better?

Post reply on HN