Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster and simpler with the command line: deep-comparing JSON files with jq

genius.engineering

Post reply on HN
Print view View original

11–20 of 93 posts

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#11

Post by vidarh »

proyb2 wrote: post #7

I'm curious how well could Crystal language handles that huge amount of JSON since most of the Ruby code could be ported over to Crystal. It has a JSON pull parser to minimize memory usage which is useful for memory constraint environment but at the expense of less performant. If that could be split up with fork Crystal processes, I believe it's feasible.

There are stream parsers for JSON for Ruby too, including bindings for C libraries like YAJL - using the default JSON parser is an awful choice for doing comparisons like that given the massive overhead of the amount of objects it'll be creating for no good reason.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#12

Post by JulianMorrison »

CyberMew wrote: post #9

I wonder if a C/C++ programm would perform better?

jq is a C program.

In theory a truly specific program could work better. In practise, the broad scope of jq allows you to discover the operations you need and respond to changes in requirements without being locked into custom code, and any given programmer probably couldn't do the same job better.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#13

Post by proyb2 »

vidarh wrote: post #8
baldeagle wrote: post #3

Earlier quoted context omitted.

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

Not parse them into a tree, to start with. Use a streaming JSON parser, and compare them token by token unless/until they diverge, at which point you take whatever actual suitable to identify the delta. Parsing it into a tree may be necessary if you want to do more complex comparisons (such as sorting child objects etc.), but even then depending on your need you may well be better off storing offsets into the file de…

I believe this comparison benchmark could be useful for you and you can expand further with more tests. Although I got downvoted for sharing a link.

https://github.com/kostya/benchmarks/blob/master/README.md

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#14

Post by gyrgtyn »

baldeagle wrote: post #3
jb3689 wrote: post #2

It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I still don't think Ruby is the best choice for doing raw JSON parsing. Last time I had to care about JSON speed we were transforming billions of events and the Ruby JSON lib was becoming a bottleneck

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

nodejs comes to mind!

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#15

Post by proyb2 »

vidarh wrote: post #11
proyb2 wrote: post #7

I'm curious how well could Crystal language handles that huge amount of JSON since most of the Ruby code could be ported over to Crystal. It has a JSON pull parser to minimize memory usage which is useful for memory constraint environment but at the expense of less performant. If that could be split up with fork Crystal processes, I believe it's feasible.

There are stream parsers for JSON for Ruby too, including bindings for C libraries like YAJL - using the default JSON parser is an awful choice for doing comparisons like that given the massive overhead of the amount of objects it'll be creating for no good reason.

Agree, I believe the benchmark shown Yaji and jq in this repo is useful for you

https://github.com/kostya/benchmarks/blob/master/README.md

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#16

Post by johnjohnsmith »

My problem with this article is the entire strategy for delivering data. A JSON file? That's probably the worst way I could think of: a CSV file would have been better. Part of the reason why databases exist is to handle exactly the problem the author is posing. There are already tools in place in SQL databases that track diffs for TB db's, and the authors could simply export patch files which would be far easier to analyze for inconsistency.

It really bothers me that the article is like "wow, check ou this awesome utility that helps us with a huge problem" instead of really thinking about "how did we get this huge problem and is there already a solution."

It just reeks of inexperience.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#17

Post by jillesvangurp »

If you have json line formatted stuff (or csv) and an aws account, you can do some nice things with Athena and SQL. We have a few simple backoffice tools that I've implemented around simple sql queries on data dumped from various systems that we have in json format. Awesome, if you want to do some quick selects, joins, etc.

If you are going to process this amount of data, don't load it all into memory and process line by line. Also do that concurrently if you have more than one CPU core available. I've done this with ruby, python, Java, and misc shell tools like jq. Use what you are comfortable with and what gets results quickly.

One neat trick with jq is to use it to convert json objects to csv and to then pipe that into csvkit for some quick and dirty sql querying. Generally gets tedious beyond a few hundred MB. I recommend switching to Athena or something similar if that becomes a regular thing for you.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#18

Post by orf »

johnjohnsmith wrote: post #16

My problem with this article is the entire strategy for delivering data. A JSON file? That's probably the worst way I could think of: a CSV file would have been better. Part of the reason why databases exist is to handle exactly the problem the author is posing. There are already tools in place in SQL databases that track diffs for TB db's, and the authors could simply export patch files which would be far easier to…

CSV is hardly an easy format to parse, or really produce. There is no CSV standard, and I bet lyrics contain all kinds of weird characters that makes choosing a separator hard.

Newline JSON is a fine interchange format for this, and the only advantage I can see for CSV is you can load it into a database in one command. Which begs the question as to why use a database at all for a simple one-off diff, when there are much more lightweight alternatives (a shell command).

So now you are converting your JSON to CSV to load it into a database to run a bunch of database diffs over it to then compare them in some way. Wouldn't that lead to the question "how did we get this huge problem and is there already a solution"?

Seems like you are the one over complicating things.

And I have to say, choosing CSV and then using a database for this task reeks of inexperience. KISS.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#19

Post by vidarh »

proyb2 wrote: post #13
vidarh wrote: post #8

Earlier quoted context omitted.

Not parse them into a tree, to start with. Use a streaming JSON parser, and compare them token by token unless/until they diverge, at which point you take whatever actual suitable to identify the delta. Parsing it into a tree may be necessary if you want to do more complex comparisons (such as sorting child objects etc.), but even then depending on your need you may well be better off storing offsets into the file de…

I believe this comparison benchmark could be useful for you and you can expand further with more tests. Although I got downvoted for sharing a link. https://github.com/kostya/benchmarks/blob/master/README.md

That still parses into a tree.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#20

Post by totally »

That's cool, you made a thing that verifies the two export jobs you wrote have the same data even though they have different output.

I can't help wondering, if you control the code that generates the JSON, why not output in a conservative, consistent format? I'm sure there are pros/cons, but this work would allow something like `diff` to work, and then you don't have to maintain a separate utility.

Post reply on HN