Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster and simpler with the command line: deep-comparing JSON files with jq

genius.engineering

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#21

Post by theamk »

baldeagle wrote: post #3
jb3689 wrote: post #2

It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I still don't think Ruby is the best choice for doing raw JSON parsing. Last time I had to care about JSON speed we were transforming billions of events and the Ruby JSON lib was becoming a bottleneck

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby

or

(2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ). Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once.

The streaming API might work better with jq-based preprocessing -- for example, if you want to compare two unsorted sets, it may be faster to sort them using jq, then compare line-by-line using streaming API.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#22

Post by mattbillenstein »

johnjohnsmith wrote: post #16

My problem with this article is the entire strategy for delivering data. A JSON file? That's probably the worst way I could think of: a CSV file would have been better. Part of the reason why databases exist is to handle exactly the problem the author is posing. There are already tools in place in SQL databases that track diffs for TB db's, and the authors could simply export patch files which would be far easier to…

You're wrong - json has types - it's very very useful just because of that, and pass newline delimited json through gzip and you basically remove all the size redundant keys...

I think this guy did the right thing for what sounded like essentially a one-off job to test this new export tool. Why would you go to all the trouble to use a SQL database for a one-off thing that can be done using text processing or worst-case writing a small script?

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#23

Post by cryptonector »

oftenwrong wrote: post #10

Another option is to use a tool like 'gron' to convert the JSON into a shell-friendly line-oriented format. This makes the rest straightforward. https://github.com/tomnomnom/gron

jq can do something like that too! Check out its --stream option.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#24

Post by cryptonector »

JulianMorrison wrote: post #12
CyberMew wrote: post #9

I wonder if a C/C++ programm would perform better?

jq is a C program. In theory a truly specific program could work better. In practise, the broad scope of jq allows you to discover the operations you need and respond to changes in requirements without being locked into custom code, and any given programmer probably couldn't do the same job better.

jq is a C program, yes, but jq programs are interpreted. Because jq is a dynamically-typed language, it wouldn't be easy to compile it to object code that would run too much faster than the byte-interpreted version (though it would still run faster).

As you say, jq's power is that it is an expressive language, and it's much much easier to write jq programs that work than it is to write C/C++ programs as needed that do the same or similar work.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#26

Post by wwarnerandrew »

jillesvangurp wrote: post #17

If you have json line formatted stuff (or csv) and an aws account, you can do some nice things with Athena and SQL. We have a few simple backoffice tools that I've implemented around simple sql queries on data dumped from various systems that we have in json format. Awesome, if you want to do some quick selects, joins, etc. If you are going to process this amount of data, don't load it all into memory and process lin…

OP here— good point! We actually use Athena to query these exports in S3 to debug data drift of specific export objects over time. It's quite a useful tool, I was able to go knowing basically nothing about Athena to querying gzipped newline-delimited JSON files in S3 using SQL in about an hour.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#27

Post by nrdgrl »

johnjohnsmith wrote: post #16

My problem with this article is the entire strategy for delivering data. A JSON file? That's probably the worst way I could think of: a CSV file would have been better. Part of the reason why databases exist is to handle exactly the problem the author is posing. There are already tools in place in SQL databases that track diffs for TB db's, and the authors could simply export patch files which would be far easier to…

It sounds like apple requires the data in JSON format - they may not have a choice.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#28

Post by nicoburns »

baldeagle wrote: post #3
jb3689 wrote: post #2

It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I still don't think Ruby is the best choice for doing raw JSON parsing. Last time I had to care about JSON speed we were transforming billions of events and the Ruby JSON lib was becoming a bottleneck

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

Rust is absolutely wonderful for tasks like this. They don't hit any of the cases where Rust's ownership can make things tricky. And the serde library makes deserializing JSON a piece of cake.

You end up with code which looks pretty similar to the equivalent JavaScript or Python code, but performs much faster (10x, 100x or even 1000x faster).

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#29

Post by nicoburns »

nicoburns wrote: post #28
baldeagle wrote: post #3

Earlier quoted context omitted.

What do you think is the best way to do deep JSON comparisons? We work with 2GB JSONs all day, and it is super annoying how long they take to process.

Rust is absolutely wonderful for tasks like this. They don't hit any of the cases where Rust's ownership can make things tricky. And the serde library makes deserializing JSON a piece of cake. You end up with code which looks pretty similar to the equivalent JavaScript or Python code, but performs much faster (10x, 100x or even 1000x faster).

There's also pikkr (https://github.com/pikkr/pikkr) if you need really really fast JSON parsing.

Re: Faster and simpler with the command line: deep-comparing JSON files with jq

#30

Post by paulddraper »

jb3689 wrote: post #2

It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib. I still don't think Ruby is the best choice for doing raw JSON parsing. Last time I had to care about JSON speed we were transforming billions of events and the Ruby JSON lib was becoming a bottleneck

> It's worth mentioning that there are much faster JSON parsing libraries than the default in Ruby stdlib.

I am on the edge of my seat now.

Would you mind listing which libraries are much (say, an order of magnitude) faster?

Post reply on HN