Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a high performance JSON parser

dave.cheney.net

Post reply on HN
Print view View original

51–60 of 193 posts

Re: Building a high performance JSON parser

#51

Post by vlovich123 »

janmo wrote: post #32
arun-mani-j wrote: post #16

I remember reading a SO question which asks for a C library to parse JSON. A comment was like - C developers won't use a library for JSON, they will write one themselves. I don't know how "true" that comment is but I thought I should try to write a parser myself to get a feel :D So I wrote one, in Python - https://arunmani.in/articles/silly-json-parser/ It was a delightful experience though, writing and testing to br…

I wrote a small JSON parser in C myself which I called jsoncut. It just cuts out a certain part of a json file. I deal with large JSON files, but want only to extract and parse certain parts of it. All libraries I tried parse everything, use a lot of RAM and are slow. Link here, if interested to have a look: https://github.com/rgex/jsoncut

The words you’re looking for are SAX-like JSON parser or streaming json parser. I don’t know if there’s any command line tools like the one you wrote that use it though to provide a jq-like interface.

Re: Building a high performance JSON parser

#52

Post by explaininjs »

gurrasson wrote: post #50
visarga wrote: post #24

nowadays I am more interested in a "forgiving" JSON/YAML parser, that would recover from LLM errors, is there such a thing?

The jsonrepair tool https://github.com/josdejong/jsonrepair might interest you. It's tailored to fix JSON strings. I've been looking into something similar for handling partial JSONs, where you only have the first n chars of a JSON. This is common with LLM with streamed outputs aimed at reducing latency. If one knows the JSON schema ahead, then one can start processing these first fields before the remaining data has…

See my sibling comment :)

Re: Building a high performance JSON parser

#53

Post by Thaxll »

jcelerier wrote: post #11
bsdnoob wrote: post #6

Earlier quoted context omitted.

Did you even open the article? Following is literally in first paragraph > This package offers the same high level json.Decoder API but higher throughput and reduced allocations

How does that contradict what the parent poster says? I think it's very weird to call something "high performance" when it looks like it's maybe 15-20% of the performance of a simdjson in c++. This is not "going from normal performance to high performance", this going from "very subpar" to "subpar"

Because even in C++ people don't use json simd most project use rapidjson which Go is on part with.

Re: Building a high performance JSON parser

#56

Post by riku_iki »

3cats-in-a-coat wrote: post #47
riku_iki wrote: post #35

Earlier quoted context omitted.

> "Json" and "Go" seem antithetical in the same sentence as "high performance" to me. As long as we are talking about _absolute performance_. Even just "Json" is problematic here as wire protocol for absolute performance no matter what will be programming language.

I think people say that as they give disproportional weight to the fact it's text-based, while ignoring how astoundingly simple and linear it is to write and read. The only way to nudge the needle is to start exchanging direct memory dumps, which is what ProtoBuff and the like do. But this is clearly only for very specific use.

> while ignoring how astoundingly simple and linear it is to write and read.

code maybe simple, but you have lots of performance penalties: resolving field keys, you need to construct some complicated data structures through memory allocations, which is expensive.

> to start exchanging direct memory dumps, which is what ProtoBuff and the like do

Protobuff actually is doing parsing, it is just binary format. What you describing is more like Flatbuffer.

> But this is clearly only for very specific use.

yes, specific use is high performance computations )

Re: Building a high performance JSON parser

#57

Post by jjice »

arun-mani-j wrote: post #16

I remember reading a SO question which asks for a C library to parse JSON. A comment was like - C developers won't use a library for JSON, they will write one themselves. I don't know how "true" that comment is but I thought I should try to write a parser myself to get a feel :D So I wrote one, in Python - https://arunmani.in/articles/silly-json-parser/ It was a delightful experience though, writing and testing to br…

I guess there are only so many ways to write a JSON parser b cause one I wrote on a train in Python looks very similar!

I thought it would be nice and simple but it really was still simpler than I expected. It's a fantastic spec if you need to throw one together yourself, without massive performance considerations.

Re: Building a high performance JSON parser

#58

Post by dleeftink »

diarrhea wrote: post #26
vjerancrnjak wrote: post #9

Earlier quoted context omitted.

If your JSON always looks the same you can also do better than general JSON parsers.

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

Somewhat tangentially related, Fabian Iwand posted this regex prefix tree visualiser/generator last week [0], which may offer some inspiration for prototyping auto generated schemas.

Re: Building a high performance JSON parser

#59

Post by isuckatcoding »

mgaunard wrote: post #55

"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications

Yes but for applications where you need to do ETL style transformations on large datasets, streaming is an immensely useful strategy.

Sure you could argue go isn’t the right tool for the job but I don’t see why it can’t be done with the right optimizations like this effort.

Re: Building a high performance JSON parser

#60

Post by cbarrick »

mgaunard wrote: post #38
kevingadd wrote: post #7

I'm surprised there's no way to say 'I really mean it, inline this function' for the stuff that didn't inline because it was too big. The baseline whitespace count/search operation seems like it would be MUCH faster if you vectorized it with SIMD, but I can understand that being out of scope for the author.

Of course you can force-inline.

Obviously you can manually inline functions. That's what happened in the article.

The comment is about having a directive or annotation to make the compiler inline the function for you, which Go does not have. IMO, the pre-inline code was cleaner to me. It's a shame that the compiler could not optimize it.

There was once a proposal for this, but it's really against Go's design as a language.

https://github.com/golang/go/issues/21536

Post reply on HN