Simdjson – Parsing Gigabytes of JSON per Second
1–10 of 202 posts
Re: Simdjson – Parsing Gigabytes of JSON per Second
#2Thanks for posting. I've been working with lidar/robotic data more recently and it's nice to work with JSON directly, when the performance is good enough.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#3This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#4Is this faster than the browser’s native parsing speed I assume?
Re: Simdjson – Parsing Gigabytes of JSON per Second
#5I guess the question is, what do you parse it to? I'm guessing definitely not turning objects into std::unordered_map and arrays into std::vector or some such. So how easy it is to use the "parsed" data structure? How easy is it to add an element to some deeply nested array for example?
Re: Simdjson – Parsing Gigabytes of JSON per Second
#6> All JSON is JavaScript, but not all JavaScript is JSON
Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).
Re: Simdjson – Parsing Gigabytes of JSON per Second
#7> All JSON is JavaScript, but not all JavaScript is JSON Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).
Basically saying any valid-format JSON is valid JS as well. But JSON doesn't have any programming features (or the nice things like non-quoted keys/trailing commas)
Re: Simdjson – Parsing Gigabytes of JSON per Second
#8I guess the question is, what do you parse it to? I'm guessing definitely not turning objects into std::unordered_map and arrays into std::vector or some such. So how easy it is to use the "parsed" data structure? How easy is it to add an element to some deeply nested array for example?
I can't speak for this project, but my own for CSV files ( https://github.com/dw/csvmonkey ) provides a high level interface that allows the tokenized data to be manipulated in-place without full decoding. The interface exported in Python is that of a plain old dictionary with one added magical semantic (lazy decode on element access). The internal representation of the parse result is a simple fixed array of (ptr, size) pairs
Methods like this are used for batch search / summation where only a fraction of the parsed data is actually relevant during any particular run. You'll find similar approaches used in e.g. the row format parser of a database like MongoDB or Postgres
Re: Simdjson – Parsing Gigabytes of JSON per Second
#9I guess the question is, what do you parse it to? I'm guessing definitely not turning objects into std::unordered_map and arrays into std::vector or some such. So how easy it is to use the "parsed" data structure? How easy is it to add an element to some deeply nested array for example?
The data is put into a "ParsedJson" object: https://github.com/lemire/simdjson/blob/master/include/simdj...
Re: Simdjson – Parsing Gigabytes of JSON per Second
#10If you're working with json objects with sizes on the higher end quite often you're not going to need the entirety of them, just a small part of them. If that is the workload what then to do is simply parse as little data as possible: skip the validation, locate the relevant bits, and then start parsing, validation and all the stuff. In this optimizing the json scanner/lexer gives much greater improvement than optimizing the parser.
Though this job is trickier than it may look. The logic to extract the "relevant" bits is often dynamic or tied to user input but for the scanner/lexer to be ultrafast it has to be tightly compiled. You can try jitting but libllvm is probably too heavyweight for parsing json.