OT, but I notice it can be run by #include-ing the simdjson.cpp file. How common is this in CPP projects?
Simdjson – Parsing Gigabytes of JSON per Second
141–150 of 202 posts
Re: Simdjson – Parsing Gigabytes of JSON per Second
#142Earlier quoted context omitted.
This code in particular won’t, since it relies on a particular extension of the x86 instruction set. I don’t believe Arduino compatible chips have simd instructions, but if they do, a similar approach could be taken.
I'm not aware of any SIMD-capable Arduino chips; even when Quark was a thing, it didn't support SIMD. It's possible to do SWAR (SIMD Within A Register) tricks to try to substitute, but on a 32-bit processor (or even a 64-bit processor) I doubt our techniques would look good. In Hyperscan, my regex project, we used SWAR for simple things (character scans) but I doubt that simdjson would work well if you tried to make…
I wonder if it's possible to do something with bitslicing?
Re: Simdjson – Parsing Gigabytes of JSON per Second
#143Earlier quoted context omitted.
Any chance to have a similar thing for s-expressions? I parse GBs of them and Common Lisp reader is very slow.
Probably not too hard. It would come down to how easy it is to detect quoting conventions so you don't accidentally parse () chars in strings. JSON is medium-easy. I don't know where the canonical definition of s-expressions you're using comes from (is it just Common Lisp?) so I don't know how this works. We'd like to have some more examples of formats people care about - I'm interested in generalizing this work. So…
Yes!!! A generalization for other kinds of simple grammars would be awesome.
On another note. As a js programmer who deals with a ton of json, I would love v8 to adopt some of the tricks into their json parser.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#144Earlier quoted context omitted.
Yeah, I've done some analysis, it's creating a ton of objects to conform to the Codable protocol, and a lot of those objects are for codingPath, which is updated for basically every node in the tree. It's not a mystery, we just don't know the best way to fix it.
Is there a reason you need to use Codable? Sorry if this sounds uninformed, I haven't taken that much time to look at what you're doing exactly (I just ran https://github.com/jeremywiebe/json-performance ).
That's one of the things we're considering. But it is by far the most idiomatic way to do things in Swift. One of the alternatives we're considering is implementing the line cache (including the update protocol) in Rust, which would be a huge performance jump.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#145Earlier quoted context omitted.
Yeah, I've done some analysis, it's creating a ton of objects to conform to the Codable protocol, and a lot of those objects are for codingPath, which is updated for basically every node in the tree. It's not a mystery, we just don't know the best way to fix it.
Is there a reason you need to use Codable? Sorry if this sounds uninformed, I haven't taken that much time to look at what you're doing exactly (I just ran https://github.com/jeremywiebe/json-performance ).
No, I don’t think the project needs to use Codable. The point of that benchmark was to evaluate Codable’s performance under Swift 5. It was posed that performance was much improved. The benchmark points out that it has a little bit but not significantly.
Codable is desirable because it encodes/decides directly to strifes vs manually picking fields out of dicts.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#146This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102
Hey Raph, have you seen https://github.com/bmkor/gason ? Seems like a low-cost bridge to a high-performance C++ implementation.
Hadn't seen that particular wrapper, but if we're going to take on an FFI solution, we're more likely to use Rust for this, and implement more logic than just JSON parsing.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#147Will this work on JSON files that are larger than the available system memory? Firebase backups are huge JSON files and we haven’t found a good way to deal with them. There are some “streaming JSON parsers” that we have wrestled with but they are buggy.
Probably not. I requires a memory allocation the size of the file for parsing.
However they have the ability to build a tape out of the json and find the interesting marks. Perhaps it can be adapted to make a fast parser than only parses the relevant stuff but zooms through the large file in blocks.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#148Earlier quoted context omitted.
From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…
So is it too slow or not?
We actually do get 60fps, but JSON parsing on the Swift side takes more than its share of total CPU load, affecting power consumption among other things. So (partly to address the trolls elsewhere in the thread), the choice of JSON does not preclude fast implementation (as the existence of simdjson proves), but it does make it dependent on the language having a performant JSON implementation. I made the assumption that this would be the case, and for Swift it isn't.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#149Earlier quoted context omitted.
... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.
It seems they're getting parsing times 1,000x slower than any other parser, 10,000x slower than simdjson. The complaint is understandable, but ironic :)
These numbers are not quite right for a variety of reasons (performance measurement methodology is hard), but to do something more of an apples-to-apples comparison, it's about 50x slower than serde in Rust. That's still a lot, obviously.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#150I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.
I have in the past parsed terabytes of JSON. The specific use case was analysing archived Reddit comments. The Reddit API uses JSON, and somebody [1] runs a server that just dumps them in a file, one line of JSON per comment, and offers them for download (compressed, obviously). So now you end up with Gigabytes of small JSONs per month, and anything you do will be quickly dominated by JSON parsing time. You could sto…
The parsed format in tape.md is quite close to the flatbuffer format. Flatbuffer can encode any json file just fine. The parse time is immediate and requires no extra memory.
It’s a great way to store big json files where you only want to access a subset of data very quickly and not load the whole file into memory.