Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

11–20 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#11

Post by chubot »

avmich wrote: post #6

> All JSON is JavaScript, but not all JavaScript is JSON Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).

The JSON spec [1] never had any updates, so it couldn't have diverged.

Kudos to Douglas Crockford for keeping it simple. I wish more standards committees would take a cue from him. (Looking at ECMAScript [2] and C++.)

There's been a tremendous amount of growth and value around JSON precisely because it's so simple and easy to implement.

People complain about the lack of comments and trailing commas, but I think those are really expanding on the initial use case of JSON, and the benefit isn't worth cost of change. JSON does some things super well, other things marginally well, and some not at all, and that's working as intended.

You can always make something separate to cover those use cases, and that seems to have happened with TOML and so forth.

(I recall there was an RFC that cleaned up ambiguities in Crockford's web page, but it just clarified things. No new features were added. So JSON is still as much of a subset of JavaScript as it ever was. On the other hand, JavaScript itself has grown wildly out of control.)

[1] http://json.org/

[2] https://news.ycombinator.com/item?id=18766361

Re: Simdjson – Parsing Gigabytes of JSON per Second

#13

Post by chubot »

xfs wrote: post #10

If you're working with json objects with sizes on the higher end quite often you're not going to need the entirety of them, just a small part of them. If that is the workload what then to do is simply parse as little data as possible: skip the validation, locate the relevant bits, and then start parsing, validation and all the stuff. In this optimizing the json scanner/lexer gives much greater improvement than optimi…

I agree that's a good strategy for big JSON. Do you know of any such "lazy" parsers?

I think the problem is that to extract arbitrary keys, you really need to parse the whole thing, although you don't need to materialize nodes for the whole thing.

But if you have big JSON with a given schema, you may be able to skip things lexically. You basically need to count {} and [], while taking into account " and \ within quoted strings.

That doesn't seem too hard. I think a tiny bit of http://re2c.org/ could do a good job of it.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#14

Post by eesmith »

chubot wrote: post #11
avmich wrote: post #6

> All JSON is JavaScript, but not all JavaScript is JSON Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).

The JSON spec [1] never had any updates, so it couldn't have diverged. Kudos to Douglas Crockford for keeping it simple. I wish more standards committees would take a cue from him. (Looking at ECMAScript [2] and C++.) There's been a tremendous amount of growth and value around JSON precisely because it's so simple and easy to implement. People complain about the lack of comments and trailing commas, but I think those…

https://en.wikipedia.org/wiki/JSON#Data_portability_issues :

> Although Douglas Crockford originally asserted that JSON is a strict subset of JavaScript, his specification actually allows valid JSON documents that are invalid JavaScript. Specifically, JSON allows the Unicode line terminators U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR to appear unescaped in quoted strings, while ECMAScript 2018 and older does not.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#15

Post by aristidb »

westurner wrote: post #12

> Requirements: […] A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013, and processors from AMD starting with the Rizen)

Also noteworthy that on Intel at least, using AVX/AVX2 reduces the frequency of the CPU for a while. It can even go below base clock.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#16

Post by androidgirl »

xfs wrote: post #10

If you're working with json objects with sizes on the higher end quite often you're not going to need the entirety of them, just a small part of them. If that is the workload what then to do is simply parse as little data as possible: skip the validation, locate the relevant bits, and then start parsing, validation and all the stuff. In this optimizing the json scanner/lexer gives much greater improvement than optimi…

Depending on usecase, the JSON lines format can make this into a pretty simple task! Obviously has to fit in with one's data structure though.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#17

Post by azinman2 »

raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

What are you using? Have you tried NSJSONSerialization? It’s quite fast (am very curious how it shows in these benchmarks), but I don’t think it does the fancy Codable stuff.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#18

Post by eridius »

azinman2 wrote: post #17
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

What are you using? Have you tried NSJSONSerialization? It’s quite fast (am very curious how it shows in these benchmarks), but I don’t think it does the fancy Codable stuff.

Swift has JSONEncoder and JSONDecoder types to do Codable, though internally they have to encode to/decode from the Foundation objects that JSONSerialization produces.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#19

Post by eridius »

raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

I wrote my own Swift JSON parser quite a while ago, https://github.com/postmates/PMJSON. In my limited benchmarking it parses slower than Foundation's JSONSerialization (by a factor of 2–2.5 IIRC) but encodes faster, and my impression was most of the time was spent constructing Dictionaries, but I didn't do too much performance work on it. It might be interesting to have someone else take a crack at improving the performance.

That said, it also includes an event-based parser (called JSONDecoder), so if you want to handle events in order to decode into your own data structure and skip the intermediate JSON data structure, you might be able to get faster than JSONSerialization that way.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#20

Post by glangdale »

One of the two authors here. Happy to answer questions.

The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

Post reply on HN