Note: use python3!
https://gist.github.com/mattbillenstein/34cf2907390102ffbabd...
Faster and simpler with the command line: deep-comparing JSON files with jq
41–50 of 93 posts
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#42Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000
Yikes, that's nasty.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#43Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000
This is an artifact of JavaScript, which even as of ES6 uses IEEE 754 double-precision floats for all numeric values. jq likely uses the same implementation internally for compatibility reasons and to avoid surprises of a different kind.
See https://www.ecma-international.org/ecma-262/6.0/#sec-ecmascr...
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#44Earlier quoted context omitted.
(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby or (2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ). Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once. The streaming API might work better with jq-based prepro…
Python is fast at parsing JSON, Go had hard time to match parsing speed of it. Additionally you have PyPy to help.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#45My problem with this article is the entire strategy for delivering data. A JSON file? That's probably the worst way I could think of: a CSV file would have been better. Part of the reason why databases exist is to handle exactly the problem the author is posing. There are already tools in place in SQL databases that track diffs for TB db's, and the authors could simply export patch files which would be far easier to…
You're wrong - json has types - it's very very useful just because of that, and pass newline delimited json through gzip and you basically remove all the size redundant keys... I think this guy did the right thing for what sounded like essentially a one-off job to test this new export tool. Why would you go to all the trouble to use a SQL database for a one-off thing that can be done using text processing or worst-ca…
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#46I'm not sure why you are comparing the data to the old export instead of against a source of truth... for example what is in the upstream data source. Also why not verify using unit tests? Who is to say that the original export is valid and not the second export.
In theory, I agree! I hope the new codebase has a set of tests to validate just that.
But, in practice, you have a downstream consumer of this data format (Apple in this case..).. Validating the old and new formats are functionally identical is just as important as validating the new format matches the upstream source of truth :)
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#47I had a similar problem diffing large API responses a few months ago and implemented an automation friendly JSON schema tool. It's a great way to make a summary of the data, especially when looking for forgotten fields for example.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#48Earlier quoted context omitted.
Oh, I see: you probably didn't know that CSV formats is also means character separated values, and can actually use non-printing ASCII characters as delimiters. You didn't think I actually meant commas did you? I guess your experience with character separated value files is very limited. But my point was to illustrate if record fields are consistent you don't need a heavyweight solution like JSON... and you clearly m…
Yes, you're totally correct. Using a heavyweight solution like JSON is beyond the pale, I should use a much more lightweight approach involving a database server. Your tone is oddly superior in your reply, which is really at odds with the technical content of your messages. > if record fields are consistent This is all very confused. The issue is that the JSON fields where not consistent compared to the baseline. So…
Like a SQLite DB? Actually, why don't we just transfer stuff as SQLite DBs. Single file, built-in schema, you can index.
I mean, HDF is super-general and stuff, but it looks like SQLite would solve all the trouble with CSVs.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#49Earlier quoted context omitted.
CSV is hardly an easy format to parse, or really produce. There is no CSV standard, and I bet lyrics contain all kinds of weird characters that makes choosing a separator hard. Newline JSON is a fine interchange format for this, and the only advantage I can see for CSV is you can load it into a database in one command. Which begs the question as to why use a database at all for a simple one-off diff, when there are m…
Oh, I see: you probably didn't know that CSV formats is also means character separated values, and can actually use non-printing ASCII characters as delimiters. You didn't think I actually meant commas did you? I guess your experience with character separated value files is very limited. But my point was to illustrate if record fields are consistent you don't need a heavyweight solution like JSON... and you clearly m…
> Oh, I see: you probably didn't know that CSV formats is also means character separated values, and can actually use non-printing ASCII characters as delimiters.
That seems unnecessarily condescending. JSON can also mean Janky Serialized Object Notation, but that's not the common case.
> I guess your experience with character separated value files is very limited.
In practice, using something other than a comma is a good solution for some problems, but not others (eg transfer corruption or you know, the OP's use case).
> a heavyweight solution like JSON.
I've literally never heard that phrase, nor does it make much sense. At best it's 2 more characters for wrapping braces with existing quoted data/numbers and at worst you have to make up a new non-interchangeable format as you run into exceptions from the diff, which can affect past encodings. Sounds more involved than using JSON. shrug
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#50Another option is to use a tool like 'gron' to convert the JSON into a shell-friendly line-oriented format. This makes the rest straightforward. https://github.com/tomnomnom/gron
Oh, this sounds handy, though i can’t imagine it’d be as performant for this particular case.