"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications
Building a high performance JSON parser
71–80 of 193 posts
Re: Building a high performance JSON parser
#72Looks pretty good! Even though I've written far too many JSON parsers already in my career, it's really nice to have a reference for how to think about making a reasonable, fast JSON parser, going through each step individually.
That said, I will say one thing: you don't really need to have an explicit tokenizer for JSON. You can get rid of the concept of tokens and integrate parsing and tokenization entirely. This is what I usually do since it makes everything simpler. This is a lot harder to do with something like the rest of ECMAscript since in something like ECMAscript you wind up needing look-ahead (sometimes arbitrarily large look-ahead... consider arrow functions: it's mostly a subset of the grammar of a parenthesized expression. Comma is an operator, and for default values, equal is an operator. It isn't until the => does or does not appear that you know for sure!)
Re: Building a high performance JSON parser
#73"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications
Yes but for applications where you need to do ETL style transformations on large datasets, streaming is an immensely useful strategy. Sure you could argue go isn’t the right tool for the job but I don’t see why it can’t be done with the right optimizations like this effort.
If performance is important why would you keep large datasets in JSON format?
Re: Building a high performance JSON parser
#74The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .
The fastest json lib in Go is the one done by the company behind Tiktok.
Excellent treat vector.
Re: Building a high performance JSON parser
#75The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson . Or the MinIO port of it to Go: https://github.com/minio/simdjson-go .
The fastest json lib in Go is the one done by the company behind Tiktok.
Excellent treat vector.
Re: Building a high performance JSON parser
#76[flagged]
This is an article about optimizing a JSON parser in Go.
As always, try to remember that people usually aren't writing (or posting their talks) specifically for an HN audience. Cheney clearly has an audience of Go programmers; that's the space he operates in. He's not going to title his post, which he didn't make for HN, just to avoid a language war on the threads here.
It's our responsibility to avoid the unproductive language war threads, not this author's.
Re: Building a high performance JSON parser
#77"It’s unrealistic to expect to have the entire input in memory" -- wrong for most applications
If you're building a library you either need to explicitly call out your limits or do streaming.
I've pumped gigs of jaon data, so a streaming parser is appreciated. Plus streaming shows the author is better at engineering and is aware of the various use cases.
Memory is not cheap or free except in theory.
Re: Building a high performance JSON parser
#78Earlier quoted context omitted.
If your JSON always looks the same you can also do better than general JSON parsers.
You might also move to something other than JSON if parsing it is a significant part of your workload.
Most of the times I’ve had to deal with JSON performance issues, it involved a 3rd party API and JSON was the only option.
If you’re building something net-new and know you’ll have these problems out the gate, something other than JSON might be feasible, but the moment some other system not in the closed loop needs to work with the data, you’re back to JSON and any associated perf issues.
Re: Building a high performance JSON parser
#79In n2[1] I needed a fast tokenizer and had the same "garbage factory" problem, which is basically that there's a set of constant tokens (like json.Delim in this post) and then strings which cause allocations.
I came up with what I think is a kind of neat solution, which is that the tokenizer is generic over some T and takes a function from byteslice to T and uses T in place of the strings. This way, when the caller has some more efficient representation available (like one that allocates less) it can provide one, but I can still unit test the tokenizer with the identity function for convenience.
In a sense this is like fusing the tokenizer with the parser at build time, but the generic allows layering the tokenizer such that it doesn't know about the parser's representation.
Re: Building a high performance JSON parser
#80[flagged]
Even sticking within the confines of json there's low hanging fruit, e.g. if your data is typically like:
[{"k1": true, "k2": [2,3,4]}, {"k1": false, "k2": []}, ...]
{"k1": [true, false, ...], "k2": [[2,3,4], [], ...]}
Or if your data is 99.999% of the time just pairs of k1 and k2, turn them into tuples:
{"k1k2": [true,[2,3,4],false,[], ...]}
{"k3": 2}