Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a high performance JSON parser

dave.cheney.net

Post reply on HN
Print view View original

151–160 of 193 posts

Re: Building a high performance JSON parser

#151

Post by zlg_codes »

craigching wrote: post #85
coldtea wrote: post #82

Earlier quoted context omitted.

What line of work are you in that you've "written far too many JSON parsers already" in your career?!!!

Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.

What on Earth are you storing in JSON that this sort of performance issue becomes an issue?

How big is 'large' here?

I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware.

I'm having difficulty picturing a JSON dataset big enough to slow down modern hardware. Maybe Gentoo's portage tree if it were JSON encoded?

Re: Building a high performance JSON parser

#152

Post by romshark »

I've recently held a talk (https://youtu.be/a7VBbbcmxyQ?si=0fGVxfc4qmKMVCXk) about github.com/romshark/jscan that I've been working on. It's a performance-oriented JSON iterator / tokenizer you might want to take a look at if interested in high performance zero allocation JSON parsing in Go.

Re: Building a high performance JSON parser

#154

Post by EdwardDiego »

A person who helped me out a lot when I was learning to code wrote his own .NET JSON library because the MS provided one had a rough API and was quite slow.

His lib became the defacto JSON lib in .NET dev and naturally, MS head-hunted him.

Fast JSON is that important these days.

Re: Building a high performance JSON parser

#155

Post by cowl »

chii wrote: post #140
crabbone wrote: post #133

Earlier quoted context omitted.

Here people confidently keep repeating "streaming JSON". What do you mean by that? I'm genuinely curios. Do you mean XML SAX-like interface? If so, how do you deal with repeated keys in "hash tables"? Do you first translate JSON into intermediate objects (i.e. arrays, hash-tables) and then transform them into application-specific structures, or do you try to skip the intermediate step? I mean, streaming tokens is kin…

> If so, how do you deal with repeated keys in "hash tables"? depending on the parser, behaviour might differ. But looking at https://stackoverflow.com/questions/21832701/does-json-synta... , it seems like the "best" option is to have 'last key wins' as the resolution. This works fine under a SAX like interface in a streaming JSON parser - your 'event handler' code will execute for a given key, and a 2nd time for the…

last key wins is terrible advice and has serious security implications.

see https://bishopfox.com/blog/json-interoperability-vulnerabili... or https://www.cvedetails.com/cve/CVE-2017-12635/ for concrete examples where this treatment causes security issues.

the https://datatracker.ietf.org/doc/html/rfc7493 defines a more strict format where duplicate keys are not allowed.

Re: Building a high performance JSON parser

#157

Post by romshark »

jensneuse wrote: post #61

I've taken a very similar approach and built a GraphQL tokenizer and parser (amongst many other things) that's also zero memory allocations and quite fast. In case you'd like to check out the code: https://github.com/wundergraph/graphql-go-tools

You might also want to check out this abomination of mine: https://github.com/graph-guard/gqlscan

I've held a talk about this, unfortunately wasn't recorded. I've tried to squeeze as much out of Go as I could and I've went crazy doing that :D

Re: Building a high performance JSON parser

#158

Post by EMM_386 »

zlg_codes wrote: post #151
craigching wrote: post #85

Earlier quoted context omitted.

Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.

What on Earth are you storing in JSON that this sort of performance issue becomes an issue? How big is 'large' here? I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware. I'm having difficulty picturing a JSON dataset big enough to slow down modern hardw…

> What on Earth are you storing in JSON that this sort of performance issue becomes an issue?

I've been in the industry for a while. I've probably left more than one client site muttering "I've seen some things ...".

If it can be done, it will be done. And often in a way that shouldn't have even been considered at all.

Many times, "it works" is all that is needed. Not exactly the pinnacle of software design. But hey, it does indeed "work"!

Re: Building a high performance JSON parser

#159

Post by Edwinr95 »

zlg_codes wrote: post #151
craigching wrote: post #85

Earlier quoted context omitted.

Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.

What on Earth are you storing in JSON that this sort of performance issue becomes an issue? How big is 'large' here? I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware. I'm having difficulty picturing a JSON dataset big enough to slow down modern hardw…

I've seen people dump and share entire databases in JSON format at my job....

Re: Building a high performance JSON parser

#160

Post by nly »

zlg_codes wrote: post #151
craigching wrote: post #85

Earlier quoted context omitted.

Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.

What on Earth are you storing in JSON that this sort of performance issue becomes an issue? How big is 'large' here? I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware. I'm having difficulty picturing a JSON dataset big enough to slow down modern hardw…

I've seen tens of millions of market data events from a single day of trading encoded in JSON and used in various post-trade pipelines.
Post reply on HN