Parsing gigabytes of JSON per second
arxiv.org
Parsing gigabytes of JSON per second
1–10 of 81 posts
Re: Parsing gigabytes of JSON per second
#2(2019)
Re: Parsing gigabytes of JSON per second
#3The project described in the paper is https://simdjson.org/.
Re: Parsing gigabytes of JSON per second
#4If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of the 100s of binary variants of json-similar formats has the same capabilities and unite around it.
Re: Parsing gigabytes of JSON per second
#5If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
I largely agree, but most modern binary formats are more rigid with schemas and types.
If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
Re: Parsing gigabytes of JSON per second
#6If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
That’s an interesting thought. I am now very fascinated by what kind of data would be gigs in size but would need the flexibility of JSON.
Re: Parsing gigabytes of JSON per second
#7If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
JSON doesn't allow any custom type, so it is not "flexible" per se. Therefore you only need a format that supports the JSON data model and pretty much nothing else; CBOR [1] for example almost surely fits the bill.
[1] https://cbor.io/
Re: Parsing gigabytes of JSON per second
#8If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
Well json is everywhere so I think it’s mostly too late for companies to change all their internal protocols and apis to not use it. And note that a big advantage of json is that many applications can process it without a schema—you don’t need to know that the username field is a certain length, type and at a particular place. It is also easier to know if json is valid than a random binary format. The advantage of something like the algorithm here is that you can put it into your shared libraries[1] to get a speed up for free. If you spend (making up numbers) 1% of your cpu time parsing json and this speeds things up 4x (note: possibly more because other parsers get more of an advantage in micro-benchmarks from the branch predictor) then if you have sufficiently many servers you can cut your server bill by 0.75% which can be a large amount of money for the largest companies.
[1] this won’t work for all languages as the OP parses json into a flattish object where fields may be looked up rather than some language-specific data structures (like js objects or python dicts or whatever)
Re: Parsing gigabytes of JSON per second
#9If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
Your dataset could, instead of gigabytes in a single request, also be a very large number of small requests all sent to a server to handle. Each message might be easily human-readable, yet the whole system would benefit from expedient processing.
Re: Parsing gigabytes of JSON per second
#10Having some trouble figuring out how to benefit from this.
I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice.
Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used a hand-rolled analogue back then but JSONStream is actually pretty cool, check it out-
What is the use case here? data lakes?