21–30 of 193 posts
The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .
[flagged]
Exactly, it's not too hard to implement in C. The one I made never copied data, instead saved the pointer/length to the data. The user only had to Memory Map the file (or equivalent), pass that data into the parse. Only memory allocation was for the Jason nodes. This way they only paid the parsing tax (decoding doubles, etc..) if the user used that data. You hit the nail on the head
Earlier quoted context omitted.
Did you even open the article? Following is literally in first paragraph > This package offers the same high level json.Decoder API but higher throughput and reduced allocations
How does that contradict what the parent poster says? I think it's very weird to call something "high performance" when it looks like it's maybe 15-20% of the performance of a simdjson in c++. This is not "going from normal performance to high performance", this going from "very subpar" to "subpar"
nowadays I am more interested in a "forgiving" JSON/YAML parser, that would recover from LLM errors, is there such a thing?
Forgiving parsers/lexers are common in language compilers for languages like rust or C# or typescript, you may want to investigate typescript in particular since it's applicable to JSON syntax. Maybe you could repurpose their parser.
The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .
If your JSON always looks the same you can also do better than general JSON parsers.
Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…
Earlier quoted context omitted.
Did you even open the article? Following is literally in first paragraph > This package offers the same high level json.Decoder API but higher throughput and reduced allocations
How does that contradict what the parent poster says? I think it's very weird to call something "high performance" when it looks like it's maybe 15-20% of the performance of a simdjson in c++. This is not "going from normal performance to high performance", this going from "very subpar" to "subpar"
Earlier quoted context omitted.
If your JSON always looks the same you can also do better than general JSON parsers.
I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…
[1] https://github.com/omissis/go-jsonschema [2] https://github.com/pquerna/ffjson
The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .
If your JSON always looks the same you can also do better than general JSON parsers.
[flagged]
Exactly, it's not too hard to implement in C. The one I made never copied data, instead saved the pointer/length to the data. The user only had to Memory Map the file (or equivalent), pass that data into the parse. Only memory allocation was for the Jason nodes. This way they only paid the parsing tax (decoding doubles, etc..) if the user used that data. You hit the nail on the head
Having done this myself, it's a massive cheat code because your bottleneck is almost always i/o and memory mapped i/o is orders of magnitude faster than sequential calls to read().
But that said it's not always appropriate. You can have gigabytes of JSON to parse, and the JSON might be available over the network, and your service might be running on a small node with limited memory. Memory mapping here adds quite a lot of latency and cost to the system. A very fast streaming JSON decoder is the move here.