Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON in 500 lines of Rust

krish.gg

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Parsing JSON in 500 lines of Rust

#51

Post by eska »

lelanthran wrote: post #10

I once (maybe a long time ago?) made a parser for JSON by: 1. Reading the entire file into RAM. 2. Providing a `const char *get_value(const char *jstring, const char *path, ...)` function with a NULL-terminated parameter list that would return the position of the value of the key at the specified path. 3. Providing a `copy_value(const char *position)` function to copy the value at the specified position. Slow? Yup! B…

That’s not a parser though. For example the caller would still need to convert escaped unicode characters, floats and bool. I guess it’s an incomplete tokenizer?

Re: Parsing JSON in 500 lines of Rust

#52

Post by nine_k »

arp242 wrote: post #25

I guess it's faster with sudo because the regular account has some resource limits, or something? Didn't mention which system is being used (Linux? macOS? Something else?) but I can't repro on my Linux system, where performance is identical: % cargo run --release [..] Parsing speed: 103761177.00 Bytes/s Parsing speed: 103.76 MB/s Parsing speed: 0.10 GB/s % doas cargo run --release [..] Parsing speed: 105401032.21 Byt…

One run is not statistically significant %) Try running 10 times each, throw away the fastest and the slowest, indicate the average and min/max among the remaining runs.

Any random background processes might slow the system slightly during the initial run. A slightly better state of the page cache on the second run could play a role, too. Flushing the FS cache before each run might be a good idea; doing cat file.json > /dev/null could be an equally good idea.

Re: Parsing JSON in 500 lines of Rust

#53

Post by nine_k »

ModernMech wrote: post #50
Surac wrote: post #44

I have no rust knowledge let me ask - is this hard to do in 500 lines of rust? - is there a catch why a implementation in rust worth mentioning on HN? don't get me wrong im realy intested in rust.

500 lines struck me as particularly long, but I would use a combinator library like nom, and it looks like they are trying to do it from scratch.

How large is the combinator library? It should not be excessively long, for the simple thing it normally does, especially in Rust which is FP-friendly.

Re: Parsing JSON in 500 lines of Rust

#54

Post by marginalia_nu »

arp242 wrote: post #26
sagacity wrote: post #24

Regarding the 'sudo' issue: Doing a benchmark by just running an example executable is not really recommended because there's a ton of reasons why you might get differing performance. It's probably better to set up an actual benchmark using a crate like Criterion instead [0]. [0] https://github.com/bheisler/criterion.rs

It's fine for things like this where you want to get a rough performance indication to see on what order of magnitude things are at (~1MB/s vs. ~10 vs. ~50 vs. ~100). A few percent error margin is fine for that. Tools like that are to eliminate noise and variation, which is an entirely different issue. According to the article, "sudo" is about 70% faster. That has nothing to do with the benchmarking method.

It's fairly realistic for the first run to be from disk, and the second from cache in a scenario like this, and ~2x difference between the two isn't entirely unrealistic.

Re: Parsing JSON in 500 lines of Rust

#55

Post by rectalogic »

thrance wrote: post #32

Do not parse using strip_prefix and then to_string, you're allocating a new buffer after each token! You can use Cow for the result of the "parse string value" function, as most JSON strings don't have escapes you can just return a reference to that slice of the buffer, only allocating when there's actually an escape. In general when writing a parser you should strive to minimize allocations and backtracking.

I changed it to use Cow, performance improved from 121.02 MB/s to 278.66 MB/s

https://github.com/rectalogic/jsonparser

Re: Parsing JSON in 500 lines of Rust

#56

Post by ModernMech »

nine_k wrote: post #53
ModernMech wrote: post #50

Earlier quoted context omitted.

500 lines struck me as particularly long, but I would use a combinator library like nom, and it looks like they are trying to do it from scratch.

How large is the combinator library? It should not be excessively long, for the simple thing it normally does, especially in Rust which is FP-friendly.

I've seen a capable combinator lib written in about 500 lines of Rust. Probably could be shorter if minimal loc were a goal. The one I use is called "nom", it's very general and has a lot of built in functionality and options, so it's much larger than 500 lines to be sure.

Re: Parsing JSON in 500 lines of Rust

#57

Post by nwpierce »

kukkamario wrote: post #16
lelanthran wrote: post #10

I once (maybe a long time ago?) made a parser for JSON by: 1. Reading the entire file into RAM. 2. Providing a `const char *get_value(const char *jstring, const char *path, ...)` function with a NULL-terminated parameter list that would return the position of the value of the key at the specified path. 3. Providing a `copy_value(const char *position)` function to copy the value at the specified position. Slow? Yup! B…

Better option would be to parse json into Bson and then use that as the in-memory format. It uses minimal memory and is actually also fast to access without parsing into some other data structure.

That sounds a little like what I came up with a little while back. My lib can stream JSON to/from my own compact binary format that is easy to traverse/emit in C:

https://github.com/nwpierce/jsb/

Re: Parsing JSON in 500 lines of Rust

#58

Post by thrance »

rectalogic wrote: post #55
thrance wrote: post #32

Do not parse using strip_prefix and then to_string, you're allocating a new buffer after each token! You can use Cow for the result of the "parse string value" function, as most JSON strings don't have escapes you can just return a reference to that slice of the buffer, only allocating when there's actually an escape. In general when writing a parser you should strive to minimize allocations and backtracking.

I changed it to use Cow , performance improved from 121.02 MB/s to 278.66 MB/s https://github.com/rectalogic/jsonparser

Good job!
Post reply on HN