Pretty cool!
This reminds me of oboe.js: https://github.com/jimhigson/oboe.js
On-demand JSON: A better way to parse documents?
31–40 of 55 posts
Re: On-demand JSON: A better way to parse documents?
#32So they're creating a DOM-like api in front of a sax style parser and getting faster results (barring FPGA and GPU research). It's released as part of SIMDJson. I wonder if that kind of front end was done in the age of SAX parsers? Such a well-written paper.
> I wonder if that kind of front end was done in the age of SAX parsers? I though that XPath over SAX was a thing, and xslt was doing sax-like parsing, but turns out I'm wrong. Which is logical considering XPath can refer to previous nodes. That being said, it looks like there is streamable xslt in xslt 3.0, but that looks more niche
I did some automata for parsing, transformation and compression in my PhD. I think that XPath is the major failure in XML standardization, with XSLT building on this. If we had a stricter language we could easily compile much of the XML stuff and do binary XML much more extensively.
Re: On-demand JSON: A better way to parse documents?
#33Is this different from what everyone was doing with XML back in the day?
JSON has a lot more optimization that XML never got. Which I think says more about general interest in XML more than anything. Even today my experience is that XML processing varies wildly from "perfectly reasonable" to "maybe I can just do this with regex instead" even with widely used parsers.
Also XML has a number of features to care about like attributes as well as elements, and also potentially about schema. It's also needlessly verbose. Even though elements open and close in a stack there isn't a universal "close" tag. That is, if `` is always considered malformed, then why isn't the syntax simply ``?
Re: On-demand JSON: A better way to parse documents?
#34So they're creating a DOM-like api in front of a sax style parser and getting faster results (barring FPGA and GPU research). It's released as part of SIMDJson. I wonder if that kind of front end was done in the age of SAX parsers? Such a well-written paper.
SAX is a push parser, presumably this is on top of a pull API like StAX. The Jakarta JSON streaming API sort of gets at this (sort of): https://jakarta.ee/specifications/platform/9/apidocs/jakarta... The basic interface to a JSON document is something like an iterator, which lets you advance through the document, token by token, and read out values when you encounter them. So if you have an array of objects with x an…
> Wrapping a tree-like interface round incremental parsing that doesn't require eager parsing or retaining arbitrary amounts of data, and doesn't leak implementation details, sounds astoundingly hard, perhaps even impossible.
I don't think they promise this and I suspect this fails to parse some pathological but correct JSON files, eg one that starts with 50 GB of [s.
Re: On-demand JSON: A better way to parse documents?
#35Sounds similar to a technique we're using to dynamically aggregate and transform JSON. We call this package "astjson" as we're doing operations like "walking" through the JSON or "merging" fields at the AST level. We wrote about the topic and how it helped us to improve the performance of our API gateway written in Go, which makes heavy use of JSON aggregations: https://wundergraph.com/blog/astjson_high_performance_json_t...
Re: On-demand JSON: A better way to parse documents?
#36This is a real “why didn’t I think of that” moment for sure. So many systems I’ve written have profiled with most of the cpu and allocations in the JSON parser, when all it needs is a few fields. But rewriting it all in SAX is just not worth all the trouble.
[deleted]
Re: On-demand JSON: A better way to parse documents?
#37Alternatively, jsonl/ndjson. The largest parts of jsons are usually arrays, not dictionaries. So you can e.g.:
{}
{}
...
{}
{}
...
{}
Re: On-demand JSON: A better way to parse documents?
#38Is this different from what everyone was doing with XML back in the day?
JSON has a lot more optimization that XML never got. Which I think says more about general interest in XML more than anything. Even today my experience is that XML processing varies wildly from "perfectly reasonable" to "maybe I can just do this with regex instead" even with widely used parsers. Also XML has a number of features to care about like attributes as well as elements, and also potentially about schema. It'…
> That is, if `` is always considered malformed, then why isn't the syntax simply ``?
XML isn't just a structured data format where close tags always run up against each other and whitespace is insignificant. It's also a descriptive document format which is often hand-authored.
I think the argument is that the close tags being named makes those documents easier for a human author to understand. It certainly is my experience.
Re: On-demand JSON: A better way to parse documents?
#39Relevant: LEJP - libwebsockets json parser.
You specify what you're interested in and then the parser calls your callback whenever it reads the part of a large JSON stream that has your key.
https://libwebsockets.org/lws-api-doc-main/html/md_READMEs_R...
Re: On-demand JSON: A better way to parse documents?
#40Sorry, I would never use this. Before I consume any json from any source or for any purpose I validate it. Lazy loading serves no purpose if you need validation. Hint: you need validation.
You don't need to load the entire JSON object as a DOM into RAM just to validate it. Validation can easily be done using a stack and iteration with space complexity being the depth of the JSON (stack) and time being linear to the length of the object.