Building a high performance JSON parser • Hacker News Board

Re: Building a high performance JSON parser

#91

Post by explaininjs » Sun, Nov 05, 2023, 6:37 PM UTC

coldtea wrote: post #83

suzzer99 wrote: post #81

Can someone explain to me why JSON can't have comments or trailing commas? I really hope the performance gains are worth it, because I've lost 100s of man-hours to those things, and had to resort to stuff like this in package.json: "IMPORTANT: do not run the scripts below this line, they are for CICD only": true,

It can't have comments because it didn't originally had comments, so now it's too late. And it originally didn't have comments, because Douglas Cockford thought they could be abused for parsing instructions. As for not having trailing commas, it's probably a less intentional bad design choice. That said, if you want commas and comments, and control the parsers that will be used for your JSON, then use JSONC (JSON wit…

JSONC also supports trailing commas. It is, in effect, "JSON with no downsides".

TOML/Yaml always drive me batty with all their obscure special syntax. Whereas it's almost impossible to look at a formatted blob of JSON and not have a very solid understanding of what it represents.

The one thing I might add is multiline strings with `'s, but even that is probably more trouble than it's worth, as you immediately start going down the path of "well let's also have syntax to strip the indentation from those strings, maybe we should add new syntax to support raw strings, ..."

Re: Building a high performance JSON parser

#92

Post by jchw » Sun, Nov 05, 2023, 6:38 PM UTC

coldtea wrote: post #82

jchw wrote: post #72

Looks pretty good! Even though I've written far too many JSON parsers already in my career, it's really nice to have a reference for how to think about making a reasonable, fast JSON parser, going through each step individually. That said, I will say one thing: you don't really need to have an explicit tokenizer for JSON. You can get rid of the concept of tokens and integrate parsing and tokenization entirely . This…

What line of work are you in that you've "written far too many JSON parsers already" in your career?!!!

Reasons differ. C++ is a really hard place to be. It's gotten better, but if you can't tolerate exceptions, need code that is as-obviously-memory-safe-as-possible, can parse incrementally (think SAX style), off-the-shelf options like jsoncpp may not fit the bill.

Handling large documents is indeed another big one. It sort-of fits in the same category as being able to parse incrementally. That said, Go has a JSON scanner you can sort of use for incremental parsing, but in practice I've found it to be a lot slower, so for large documents it's a problem.

I've done a couple in hobby projects too. One time I did a partial one in Win32-style C89 because I wanted one that didn't depend on libc.

Re: Building a high performance JSON parser

#93

Post by querulous » Sun, Nov 05, 2023, 6:44 PM UTC

dh2022 wrote: post #73

isuckatcoding wrote: post #59

Earlier quoted context omitted.

Yes but for applications where you need to do ETL style transformations on large datasets, streaming is an immensely useful strategy. Sure you could argue go isn’t the right tool for the job but I don’t see why it can’t be done with the right optimizations like this effort.

If performance is important why would you keep large datasets in JSON format?

sometimes it's not your data

Re: Building a high performance JSON parser

#94

Post by marginalia_nu » Sun, Nov 05, 2023, 6:46 PM UTC

diarrhea wrote: post #26

vjerancrnjak wrote: post #9

Earlier quoted context omitted.

If your JSON always looks the same you can also do better than general JSON parsers.

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

A fundamental problem with JSON parsing is that it has variable length fields that don't encode their length, in a streaming scenario you basically need to keep resizing your buffer until the data fits. If the data is on disk and not streaming you may get away with reading ahead to find the end of the field first, but that's also not particularly fast.

Schemas can't fix that.

Re: Building a high performance JSON parser

#95

Post by ncruces » Sun, Nov 05, 2023, 6:49 PM UTC

It's possible to improve over the standard library with better API design, but it's not really possible to do a fully streaming parser that doesn't half fill structures before finding an error and bailing out in the middle, which is another explicit design constraint for the standard library.

Re: Building a high performance JSON parser

#96

Post by hintymad » Sun, Nov 05, 2023, 6:54 PM UTC

How is this compared to Daniel Lemire's simdjson? https://github.com/simdjson/simdjson

Re: Building a high performance JSON parser

#97

Post by tubthumper8 » Sun, Nov 05, 2023, 6:55 PM UTC

coldtea wrote: post #83

suzzer99 wrote: post #81

Can someone explain to me why JSON can't have comments or trailing commas? I really hope the performance gains are worth it, because I've lost 100s of man-hours to those things, and had to resort to stuff like this in package.json: "IMPORTANT: do not run the scripts below this line, they are for CICD only": true,

It can't have comments because it didn't originally had comments, so now it's too late. And it originally didn't have comments, because Douglas Cockford thought they could be abused for parsing instructions. As for not having trailing commas, it's probably a less intentional bad design choice. That said, if you want commas and comments, and control the parsers that will be used for your JSON, then use JSONC (JSON wit…

Does JSONC have a specification or formal definition? People have suggested[1] using JSON5[2] instead for that reason

[1] https://github.com/microsoft/vscode/issues/100688

[2] https://spec.json5.org/

Re: Building a high performance JSON parser

#98

Post by cbarrick » Sun, Nov 05, 2023, 7:16 PM UTC

mgaunard wrote: post #89

cbarrick wrote: post #60

Earlier quoted context omitted.

Obviously you can manually inline functions. That's what happened in the article. The comment is about having a directive or annotation to make the compiler inline the function for you, which Go does not have. IMO, the pre-inline code was cleaner to me. It's a shame that the compiler could not optimize it. There was once a proposal for this, but it's really against Go's design as a language. https://github.com/golang…

You can in any systems programming language. Go is mostly a toy language for cloud people.

> toy language

You may be surprised to hear that Go is used in a ton of large scale critical systems.

Re: Building a high performance JSON parser

#99

Post by jjeaff » Sun, Nov 05, 2023, 7:47 PM UTC

mannyv wrote: post #77

mgaunard wrote: post #55

"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications

If you're building a library you either need to explicitly call out your limits or do streaming. I've pumped gigs of jaon data, so a streaming parser is appreciated. Plus streaming shows the author is better at engineering and is aware of the various use cases. Memory is not cheap or free except in theory.

I guess it's all relative. Memory is significantly cheaper if you get it anywhere but on loan from a cloud provider.

Re: Building a high performance JSON parser

#100

Post by mananaysiempre » Sun, Nov 05, 2023, 8:29 PM UTC

cannonpalms wrote: post #88

ken47 wrote: post #34

Earlier quoted context omitted.

Fastest at what?

> For all sizes of json and all scenarios of usage, Sonic performs best. The repository has benchmarks

I’m not seeing simdjson in them though? I must be missing something because the Go port of it is explicitly mentioned in the motivation[1] (not the real thing, though).

[1] https://github.com/bytedance/sonic/blob/main/docs/INTRODUCTI...