Building a high performance JSON parser • Hacker News Board

Re: Building a high performance JSON parser

#191

Post by kiitos » Thu, Nov 09, 2023, 1:44 AM UTC

crabbone wrote: post #179

chii wrote: post #176

Earlier quoted context omitted.

you're layering the application semantics into the transport format. It's fine, in the sense that a JSON with duplicate keys is already invalid - but the parser might handle it, and i suggested a way (just from reading the stackoverflow answer). It's the same "fine" that you get from undefined C compiler behaviour.

Why do you keep inventing stuff... No, JSON with duplicate keys is not invalid. The whole point of streaming is to be able to process data before it completely arrived. What "layering semantics" are you talking about? This has no similarity with undefined behavior. This is documented and defined.

A JSON object with duplicate keys is explicitly defined by the spec as undefined behavior, and is left up to the individual implementation to decide what to do. It's neither valid nor invalid.

Re: Building a high performance JSON parser

#192

Post by kiitos » Thu, Nov 09, 2023, 1:47 AM UTC

cowl wrote: post #155

chii wrote: post #140

Earlier quoted context omitted.

> If so, how do you deal with repeated keys in "hash tables"? depending on the parser, behaviour might differ. But looking at https://stackoverflow.com/questions/21832701/does-json-synta... , it seems like the "best" option is to have 'last key wins' as the resolution. This works fine under a SAX like interface in a streaming JSON parser - your 'event handler' code will execute for a given key, and a 2nd time for the…

last key wins is terrible advice and has serious security implications. see https://bishopfox.com/blog/json-interoperability-vulnerabili... or https://www.cvedetails.com/cve/CVE-2017-12635/ for concrete examples where this treatment causes security issues. the https://datatracker.ietf.org/doc/html/rfc7493 defines a more strict format where duplicate keys are not allowed.

Last key wins is the most common behavior among widely-used implementations. It should be assumed as the default.

Re: Building a high performance JSON parser

#193

Post by pbrumm » Thu, Nov 09, 2023, 1:44 PM UTC

zlg_codes wrote: post #151

craigching wrote: post #85

Earlier quoted context omitted.

Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.

What on Earth are you storing in JSON that this sort of performance issue becomes an issue? How big is 'large' here? I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware. I'm having difficulty picturing a JSON dataset big enough to slow down modern hardw…

Insurance price transparency can have 16gb of compressed JSON that represents a single object.

Here is the anthem page. The toc link is 16gb

https://www.anthem.com/machine-readable-file/search/

They are complying with the mandate. But not optimizing for the parsers