31–40 of 59 posts
1. https://github.com/ezpkg/iter.json: last year, using go iterator, the core parsing code [1a] is around 200 lines
1a. https://github.com/ezpkg/ezpkg/blob/main/iter.json/parser.go
2. https://github.com/iOliverNguyen/ujson: 4 years ago, using callback, around 400 lines
In general when writing a parser you should strive to minimize allocations and backtracking.
I once (maybe a long time ago?) made a parser for JSON by: 1. Reading the entire file into RAM. 2. Providing a `const char *get_value(const char *jstring, const char *path, ...)` function with a NULL-terminated parameter list that would return the position of the value of the key at the specified path. 3. Providing a `copy_value(const char *position)` function to copy the value at the specified position. Slow? Yup! B…
Only safe and easy if json file fits into the RAM.
So?
Many of the other JSON parsing tools build up a tree in RAM[1], so the statement "Only works if the final tree fits in RAM" is just as true, and since JSON is mostly text, the final tree is not that much smaller than the source JSON anyway.
[1] Including the one this article is presenting, and all the other solutions presented in the comments.
I once (maybe a long time ago?) made a parser for JSON by: 1. Reading the entire file into RAM. 2. Providing a `const char *get_value(const char *jstring, const char *path, ...)` function with a NULL-terminated parameter list that would return the position of the value of the key at the specified path. 3. Providing a `copy_value(const char *position)` function to copy the value at the specified position. Slow? Yup! B…
Better option would be to parse json into Bson and then use that as the in-memory format. It uses minimal memory and is actually also fast to access without parsing into some other data structure.
Here is one in 73 lines of F# by Jon Harrop: https://gist.github.com/isaksky/6681cfad8ced1708a04b2eca92fc...
match object(src) { Ok(res) => return Ok(res), Err(JSONParseError::NotFound) => {} // if not found, that ok Err(e) => return Err(e), } You probably have realized that this is really tedious, and this is where macros would really shine: macro_rules! try_parse_as { ($f:expr) => ( match $f(src) { Ok(res) => return Ok(res), Err(JSONParseError::NotFound) => {} // if not found, that ok Err(e) => return Err(e), } ); } try_p…
> Fortunately `f64::parse` accepts a strict superset of JSON number grammar Just for the sake of completeness, and not to imply that you don't know this, but the JSON spec doesn't limit the size or precision of numbers, although it allows implementations set other limits. I have encountered JSON documents that (annoyingly) required the use of a parser with bigint-support.
This actually led to a data-loss causing bug in the AWS DynamoDB Console's editor a couple of years ago. IIRC, the temporary fix was to fail if the input-number couldn't be represented by a 64-bit float. Can't remember if it ever got a proper fix.
Parsing JSON is a Minefield: https://seriot.ch/projects/parsing_json.html
I regularly deal with situations where devs "log" by sending "json" to stdout of their container runtime, then expect the downstream infrastructure to magic structure into it perfectly. "You understand something has to parse that stream of bytes you're sending to find all the matching quotes and curly braces and stuff, right? What happens if one process in the container's emitting a huge log event and something else in the container decides to report that it's doing some memory stuff?" "I expect you'd log the error and open an incident?"
(the correct answer is to just collect the garbage strings in json (ha) and give them unparsed crap for them to deal with themselves, but then "we're devs deving; we don't want to waste energy on operations toil"
Later people ask "why's logging so expensive?"
Sigh.
[1] opensearch / elasticsearch, obvs
Now try parsing Rust in 500 lines of JSON
I guess it's faster with sudo because the regular account has some resource limits, or something? Didn't mention which system is being used (Linux? macOS? Something else?) but I can't repro on my Linux system, where performance is identical: % cargo run --release [..] Parsing speed: 103761177.00 Bytes/s Parsing speed: 103.76 MB/s Parsing speed: 0.10 GB/s % doas cargo run --release [..] Parsing speed: 105401032.21 Byt…
Here is one in 73 lines of F# by Jon Harrop: https://gist.github.com/isaksky/6681cfad8ced1708a04b2eca92fc...
https://fsharpforfunandprofit.com/posts/understanding-parser...