Building a high performance JSON parser • Hacker News Board

Re: Building a high performance JSON parser

#31

Post by Thaxll » Sun, Nov 05, 2023, 3:13 PM UTC

eatonphil wrote: post #3

The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .

The fastest json lib in Go is the one done by the company behind Tiktok.

Re: Building a high performance JSON parser

#32

Post by janmo » Sun, Nov 05, 2023, 3:15 PM UTC

arun-mani-j wrote: post #16

I remember reading a SO question which asks for a C library to parse JSON. A comment was like - C developers won't use a library for JSON, they will write one themselves. I don't know how "true" that comment is but I thought I should try to write a parser myself to get a feel :D So I wrote one, in Python - https://arunmani.in/articles/silly-json-parser/ It was a delightful experience though, writing and testing to br…

I wrote a small JSON parser in C myself which I called jsoncut. It just cuts out a certain part of a json file. I deal with large JSON files, but want only to extract and parse certain parts of it. All libraries I tried parse everything, use a lot of RAM and are slow.

Link here, if interested to have a look: https://github.com/rgex/jsoncut

Re: Building a high performance JSON parser

#33

Post by RichieAHB » Sun, Nov 05, 2023, 3:18 PM UTC

visarga wrote: post #24

nowadays I am more interested in a "forgiving" JSON/YAML parser, that would recover from LLM errors, is there such a thing?

I feel like trying to infer valid JSON from invalid JSON is a recipe for garbage. You’d probably be better off doing a second pass with the “JSON” through the LLM but, as the sibling commenter said, at this point even the good JSON may be garbage …

Re: Building a high performance JSON parser

#34

Post by ken47 » Sun, Nov 05, 2023, 3:36 PM UTC

Thaxll wrote: post #31

eatonphil wrote: post #3

The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .

The fastest json lib in Go is the one done by the company behind Tiktok.

Fastest at what?

Re: Building a high performance JSON parser

#35

Post by riku_iki » Sun, Nov 05, 2023, 3:39 PM UTC

Galanwe wrote: post #5

[flagged]

> "Json" and "Go" seem antithetical in the same sentence as "high performance" to me. As long as we are talking about _absolute performance_.

Even just "Json" is problematic here as wire protocol for absolute performance no matter what will be programming language.

Re: Building a high performance JSON parser

#36

Post by mannyv » Sun, Nov 05, 2023, 3:42 PM UTC

These are always interesting to read because you get to see runtime quirks. I'm surprised there was so much function call overhead, for example. And it's interesting you can bypass range checkong.

The most important thing, though, is the process: measure then optimize.

Re: Building a high performance JSON parser

#37

Post by lionkor » Sun, Nov 05, 2023, 3:53 PM UTC

eatonphil wrote: post #3

The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .

simdjson has not been the fastest for a long long time

Re: Building a high performance JSON parser

#38

Post by mgaunard » Sun, Nov 05, 2023, 3:57 PM UTC

kevingadd wrote: post #7

I'm surprised there's no way to say 'I really mean it, inline this function' for the stuff that didn't inline because it was too big. The baseline whitespace count/search operation seems like it would be MUCH faster if you vectorized it with SIMD, but I can understand that being out of scope for the author.

Of course you can force-inline.

Re: Building a high performance JSON parser

#39

Post by galangalalgol » Sun, Nov 05, 2023, 3:57 PM UTC

diarrhea wrote: post #26

vjerancrnjak wrote: post #9

Earlier quoted context omitted.

If your JSON always looks the same you can also do better than general JSON parsers.

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

Doesn't the serde crate's json support do precisely this? It generates structs that have optional in all the right places and with all the right types anyway. Seems like the llvm optimiser can probably do something useful with that even if the serde feature isn't using apriori knowledge out of the schema.

Re: Building a high performance JSON parser

#40

Post by masklinn » Sun, Nov 05, 2023, 4:00 PM UTC

arun-mani-j wrote: post #16

I remember reading a SO question which asks for a C library to parse JSON. A comment was like - C developers won't use a library for JSON, they will write one themselves. I don't know how "true" that comment is but I thought I should try to write a parser myself to get a feel :D So I wrote one, in Python - https://arunmani.in/articles/silly-json-parser/ It was a delightful experience though, writing and testing to br…

> I remember reading a SO question which asks for a C library to parse JSON. A comment was like - C developers won't use a library for JSON, they will write one themselves.

> I don't know how "true" that comment is

Either way it's a good way to get a pair of quadratic loops in your program: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...