51–60 of 93 posts
Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000
Yikes, that's nasty.
> My first thought was to write a ruby script to parse and compare the two exports, but after spending a little time coding something up I had a program that was starting to get fairly complicated, didn't work correctly, and was too slow—my first cut took well over an hour. Then I thought: is this one of those situations where a simple series of shell commands can replace a complex purpose-built script? Key takeaway:…
That's cool, you made a thing that verifies the two export jobs you wrote have the same data even though they have different output. I can't help wondering, if you control the code that generates the JSON, why not output in a conservative, consistent format? I'm sure there are pros/cons, but this work would allow something like `diff` to work, and then you don't have to maintain a separate utility.
good question! the analysis that I was doing was really a one-off for switching between these processes. We have unit tests and sanity checks to ensure consistency going forward, but as a final check before flipping the switch we wanted to be as confident as possible that we hadn't introduced any regressions across the full data-set. The new export process is much more reliable and a _lot_ faster, but as a side effec…
Btw I'm surprised you needed -M, since I thought jq would suppress colors if it saw it wasn't writing to a tty.
Earlier quoted context omitted.
It is. But it's a problem of JSON itself, not just jq.
JSON != JavaScript > echo 1152921504606846976 | python -c 'import sys, json; print(json.load(sys.stdin))' 1152921504606846976
A much simpler way to do this is simply to hash the files, for example using sha256sum, which AFAIK ships with just about every Linux distro. Then just compare the hashes.
Earlier quoted context omitted.
It is. But it's a problem of JSON itself, not just jq.
JSON != JavaScript > echo 1152921504606846976 | python -c 'import sys, json; print(json.load(sys.stdin))' 1152921504606846976
Earlier quoted context omitted.
It is. But it's a problem of JSON itself, not just jq.
JSON != JavaScript > echo 1152921504606846976 | python -c 'import sys, json; print(json.load(sys.stdin))' 1152921504606846976