Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

171–180 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#171

Post by kragen »

marton78 wrote: post #49
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

After spending most of a year doing deep surgery on systems that used CBOR extensively, I can report that the common CBOR parsers are not faster than common JSON parsers; surprisingly, they are actually slower. CBOR is also not easier; it's much less widely supported, and you need a separate debugging representation. It does have three real advantages over JSON: it supports binary strings, it's a monument to Carsten Bormann's ego, and data encoded in CBOR takes slightly fewer bytes than the same data encoded in JSON. (The second is only an advantage if you're Carsten Bormann.)

Re: Simdjson – Parsing Gigabytes of JSON per Second

#172

Post by greglindahl »

anderspitman wrote: post #157
aratno wrote: post #53

Earlier quoted context omitted.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

I'm going off topic at this point but I'd think for a native app the main advantages of a binary format would be the static typing and code generation that come from using an IDL.

You can have a binary format that's self-describing. It's important to understand all of the independent parts that go into a format.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#173

Post by Erwin »

kitd wrote: post #141

OT, but I notice it can be run by #include -ing the simdjson.cpp file. How common is this in CPP projects?

It seems like there are quite a few single-header C++ libraries: https://github.com/nothings/single_file_libs

The people complaining about dependency management in Python should try doing it in C++; there seems to be half a dozen competing ones. And three times as many build systems.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#174

Post by maliker »

If this kind of work is interesting to you, you might like Daniel Lemire's blog (https://lemire.me/blog/).

He's a professor, but his work is highly applied and immediately usable. He manages to find and demonstrate a lot of code where we assume the big-O performance, but the reality of modern processors and caching (etc.) mean very difference performance in practice.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#175

Post by gritzko »

mpweiher wrote: post #88
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

Yeah, Swift-most-everything is pretty slow, but particularly parsing/generating. Pre-Swift Foundation serialisation code was already...majestic, and in the Swift conversion they've typically managed to slow things down even further. Which didn't seem possible, but they managed. I have given a bunch of talks[1] on this topic, there's also a chapter in my iOS/macOS performance book[2], which I really recommend if you w…

That resonates well with my conclusions that led to the Replicated Object Notation project. [1]. If the parser creates an AST tree or some number of dictionaries or some other bullshit... "now you have two problems", that's it.

I settled on a tabular-log format, which is streamed and immediately consumed most of the time, no intermediate object structures.

Then, that "text vs binary" distinction became mostly moot. The binary is slightly more efficient, but grossly less readable, so no big gain, unless at grand scale.

[1] http://replicated.cc

Re: Simdjson – Parsing Gigabytes of JSON per Second

#176

Post by glangdale »

floofy222 wrote: post #156
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I see that you are using MMX intrinsics directly, like _mm_sub_pi8, but you are never calling _mm_empty ( https://software.intel.com/sites/landingpage/IntrinsicsGuide... ) as required by the SysV AMD64 ABI (and pretty much all other ABIs out there). I think the behavior of all the code that touches is undefined (it breaks the calling convention of the ABI), and while this often results in corrupted floating point val…

I would be extremely surprised if we were somehow accidentally using MMX; it's not our intention. It is my belief that we are using only AVX2, which, like the 19-year old SSE/SSE2 extension, has its own registers that are independent of the x87 floating point set.

If, once you review our codebase and verify that we are not inadvertently using a 22-year-old SIMD extension but still have undefined behavior, please write an issue on github.

I'm admiring Rust from a distance at this stage. I am comfortable enough with writing bare intrinsics and slapping a giant #ifdef around stuff.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#177

Post by spc476 »

kragen wrote: post #171
marton78 wrote: post #49

Earlier quoted context omitted.

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

After spending most of a year doing deep surgery on systems that used CBOR extensively, I can report that the common CBOR parsers are not faster than common JSON parsers; surprisingly, they are actually slower. CBOR is also not easier; it's much less widely supported, and you need a separate debugging representation. It does have three real advantages over JSON: it supports binary strings, it's a monument to Carsten…

There are a few more advantages to CBOR:

1) there's a distinction between integers and floating point values;

2) you can semantically tag values (yes, this is a text string, but treat it as a date; this is a binary string, but treat it as a big number; etc.);

3) you can have maps with non-text keys.

I'm not sure what Carsten Bormann's ego has to do with CBOR, but I found RFC-7049 one of the better written specs, with plenty of encoding examples. It made it real easy to write a encoder/decoder [1] and use the examples as test cases.

[1] https://github.com/spc476/CBOR

Re: Simdjson – Parsing Gigabytes of JSON per Second

#178

Post by glangdale »

billconan wrote: post #166
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I recently saw people using gpu to parse csv files. there are also other articles on using gpu to parse json. do you think if gpu can perform well on this type of tasks?

I'm not aware of an article that covers a actual implementation or that has a benchmark of performance. As for GPGPU: it's possible. Our first stage of matching is very parallel. But Amdahl's law would, of course, suggest that the serial parsing step would dominate.

I'm interested in this: some aspects of our very serial 'stage 2' (the parsing step) could be made parallel. This would be very interesting. Unfortunately I personally cannot be made parallel, so working on this needs to go into a big queue with a lot of other work.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#179

Post by glangdale »

jackdh wrote: post #140
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

Why did you decide to write this? What was the motivation?

Honestly? I was trolled into it. :-) Unemployed people do weird things.

I can't speak for Daniel's motivation.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#180

Post by glangdale »

ftp-bit wrote: post #132

Perhaps I'm misunderstanding or don't have a good enough grasp of this, but, in what circumstance would you need to parse gigabytes? I've only seen it be used in config files, so...

There are some quite big JSON files out there; you might also be interested in parsing megabytes but not spending more than 1ms to get through it.
Post reply on HN