I wanted to like jq, but honestly, I can't figure out it's crazy syntax.
it's a bit weird, but it's perfect for writing quick shell one liners with once you get used to it. no regular scripting language can match it for that
Faster and simpler with the command line: deep-comparing JSON files with jq
71–80 of 93 posts
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#72I love jq.
I replaced a bunch of bespoke ETL code with shell scripts. grep, sed, jq, xsv, psql, etc. Fast, efficient, iterative, inspectable, portable.
Alas, most everyone else insists on python, nodejs, ruby, AWS Lambda, jenkins goo, misc mayfly tech stacks. So my "use the most simple tool that works" advocacy has never gained traction.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#73Earlier quoted context omitted.
Python's json package != JSON JSON: https://tools.ietf.org/html/rfc8259#page-8
The link says that it's up to the implementation, which means it's valid for Python's JSON implementation to support larger numbers. It's less "interoperable" but not strictly invalid, by my read.
What the GP means is that JSON doesn't require an implementation to decode JSON integers as arbitrary-precision integers, to be "conformant JSON."
Therefore, you can't assume that if you pass some JSON through an arbitrary pipeline of JSON-manipulating tools, written in various languages, that your integer values will be passed through losslessly.
Therefore, you just shouldn't use JSON integer values when you know that the values can potentially be large. This is why e.g. Ethereum's JSON-RPC API uses hex-escaped strings (e.g. "0x0") for representing its "quantity" type.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#74+1 for a great tool.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#75This is somewhat related to a hack I threw together recently: https://github.com/ecordell/jf
It attempts to address a similar problem (comparing json or subsets of json), but I wanted the structure of what was being compared to be more readable (compared to jq), so I went with graphql syntax. Doubt it would do great on larger datasets though.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#76Earlier quoted context omitted.
Yes, you're totally correct. Using a heavyweight solution like JSON is beyond the pale, I should use a much more lightweight approach involving a database server. Your tone is oddly superior in your reply, which is really at odds with the technical content of your messages. > if record fields are consistent This is all very confused. The issue is that the JSON fields where not consistent compared to the baseline. So…
Like a SQLite DB? Actually, why don't we just transfer stuff as SQLite DBs. Single file, built-in schema, you can index. I mean, HDF is super-general and stuff, but it looks like SQLite would solve all the trouble with CSVs.
I'm currently implementing an "individual-scale data-warehouse" service (i.e. "Hadoop without the Hadoop part"), and I'm currently pondering between the choices of "a tarball of CSVs", an SQLite file, and an Apache Avro file, as input-side wire formats. (And now HDF5 as well; didn't know about that one.)
I'm still leaning toward "a tarball of CSVs", though:
1. it's very easy to allow different devs to write a bunch of single-purpose Extract tools, each in whatever language is best for the job (e.g. Python if it has to use an API where the only available API-client library impl is in Python) to scrape some particular dimension out of an external source. You can write out CSV data in pretty much any language—even a bash script! That's because, even if the language doesn't have a CSV library, a "trivial" CSV dump can be accomplished by just calling printf(2) with a CSV template string. (Trivial = you know all your stringly-typed data is of constrained formats, such that it doesn't require any quoting. CSV files are trivial more often than you'd think!)
2. Presuming your CSV file is column-ordered to have any primary key(s) first, and that it has no embedded header line, you can "reduce" on the output of a bunch of Extract jobs (i.e. merge-sorting + deduping to produce one CSV dataset) by just feeding all the input files to sort(1) with the -u and -n switches passed. `sort -n -t ','` basically behaves as a very simple streaming CSV parser, while also being amazingly-well-optimized at chewing through on-disk files in parallel. sort(1) is to (local on-disk) CSV data as LevelDB's compaction algorithm is to key-value pair data: a solid primitive that scales with your dataset size.
3. Once you've got two sorted+deduped CSV "snapshot" files, you can create a differential snapshot from them just by calling:
comm -1 -3 "$old_csv" "$new_csv" > diff.csv
cat pre_ddl.sql copy_stmt_begin.sql header.csv diff.csv copy_stmt_end.sql post_ddl.sql > migration.sql
That being said, the other formats are nice for 1. keeping data like numbers in more compact binary forms, 2. being able to sort and de-dup the data slightly more cheaply, without having to parse anything at point-of-sort. (Though this matters less than you'd think; sort(1)'s minimal parser is very fast, and SQLite/Avro can't get any big access-time wins since the data is neither pre-sorted nor column-oriented.)
But in exchange for this, you lose the ability to cheaply merge working datasets together. You can't just concatenate them—you have to ask your storage-format library to serialize data from one of your data files, and then ask the library to parse and import said data into your other data file. Frequently, the overhead of a complete deep parse of the data is the thing we're trying to avoid the expense of in the first place! (Otherwise, why use an ETL pipeline at all, when you could just have your operational data sources do batched SQL inserts directly to your data warehouse?)
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#77Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#78Earlier quoted context omitted.
it's a bit weird, but it's perfect for writing quick shell one liners with once you get used to it. no regular scripting language can match it for that
awk, sed?
For extracting a value out of json where the keys can arbitrarily re-order and you have nested maps containing the same key names? No thanks.
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#79As part of an automated Jira upgrade script (well, Makefile) we needed to export changes to the listener port/scheme configuration which is unfortunately stored “in the code” so to speak (in WEB-INF/web.xml) which Atlassian doesn’t deign to whitespace normally (indentation is all over the place, as is formatting, character encoding, and more) —- and they mangle it differently somehow with each point release. So the Makefile calls xmllint to normalize formatting and whitespace of both the untouched source files from the old and new release as well as the locally modified (deployed) configuration, then calls diff/patch accordingly (in a three-way).
Re: Faster and simpler with the command line: deep-comparing JSON files with jq
#80Just a heads up to anyone using jq - I've previously spent a couple of hours debugging a problem because jq uses float64 to store integers (which might lead to rounding-errors/overflows). For example: echo 1152921504606846976 | jq 1152921504606847000
Not jq but JS itself.
Here's the epic jwz rant on it: https://www.jwz.org/blog/2010/10/every-day-i-learn-something...
with Brendan Eich chiming in: https://www.jwz.org/blog/2010/10/every-day-i-learn-something...