Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

51–60 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#51

Post by dmix »

oh_sigh wrote: post #46
adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

What if you're ingesting thousands or millions of small feeds? You might not have much control or desire to dictate format to your clients

Yeah not everyone, I’d even say the majority of people, are using software parsing libraries where they are in control of the input data format.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#52

Post by Skinney »

marton78 wrote: post #49
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

Because JSON encoding/decoding was not found to be a typical performance bottleneck, and because JSON is supported in virtually every programming language (Xi allows you to write frontends in pretty much any language you want).

Re: Simdjson – Parsing Gigabytes of JSON per Second

#53

Post by aratno »

marton78 wrote: post #49
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

From “Design Decisions”[1]:

> JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones.

1: https://github.com/xi-editor/xi-editor/blob/master/README.md...

Re: Simdjson – Parsing Gigabytes of JSON per Second

#54

Post by empyrical »

eesmith wrote: post #14
chubot wrote: post #11

Earlier quoted context omitted.

The JSON spec [1] never had any updates, so it couldn't have diverged. Kudos to Douglas Crockford for keeping it simple. I wish more standards committees would take a cue from him. (Looking at ECMAScript [2] and C++.) There's been a tremendous amount of growth and value around JSON precisely because it's so simple and easy to implement. People complain about the lack of comments and trailing commas, but I think those…

https://en.wikipedia.org/wiki/JSON#Data_portability_issues : > Although Douglas Crockford originally asserted that JSON is a strict subset of JavaScript, his specification actually allows valid JSON documents that are invalid JavaScript. Specifically, JSON allows the Unicode line terminators U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR to appear unescaped in quoted strings, while ECMAScript 2018 and older doe…

That bit of incompatibility will be going away when this proposal is implemented, however:

https://github.com/tc39/proposal-json-superset

Re: Simdjson – Parsing Gigabytes of JSON per Second

#55

Post by sitkack »

scottlamb wrote: post #29
aristidb wrote: post #15

Earlier quoted context omitted.

Also noteworthy that on Intel at least, using AVX/AVX2 reduces the frequency of the CPU for a while. It can even go below base clock.

iirc, it's complicated. Some instructions don't reduce the frequency; some reduce it a little; some reduce it a lot. I'm not sure AVX2 is as ubiquitous as the README says: "We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel." I guess "mainstream" is somewhat subjective, but some recent Chromebooks have Celeron processors with no AVX2: https://us-store.acer.com/…

AVX2 also incurs some pretty large penalties for switching between SSE and AVX2. Depending on the amount of time taken in the library between calls, it could be problematic.

This looks mostly applicable to server scenarios where the runtime environment is highly controlled.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#56

Post by captbaritone »

oh_sigh wrote: post #46
adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

What if you're ingesting thousands or millions of small feeds? You might not have much control or desire to dictate format to your clients

The readme specifies that it’s not optimized for reading a large number of small files.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#57

Post by tobilg »

Jerry2 wrote: post #45
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

We‘re using AWS Kinesis delivery streams to batch incoming JSON messages from IoT devices to Parquet files in S3. Those can directly be read by different AWS services like Redshift, EMR or Athena...

Re: Simdjson – Parsing Gigabytes of JSON per Second

#58

Post by saurik »

aratno wrote: post #53
marton78 wrote: post #49

Earlier quoted context omitted.

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#59

Post by defen »

glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

How hard would it be to extend the parser to handle arbitrary-precision numbers? Strictly speaking the JSON spec does not require numbers to fit into 64-bit ints / doubles.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#60

Post by glangdale »

abhorrence wrote: post #31
achalkley wrote: post #30

With this work on an Arduino?

This code in particular won’t, since it relies on a particular extension of the x86 instruction set. I don’t believe Arduino compatible chips have simd instructions, but if they do, a similar approach could be taken.

I'm not aware of any SIMD-capable Arduino chips; even when Quark was a thing, it didn't support SIMD.

It's possible to do SWAR (SIMD Within A Register) tricks to try to substitute, but on a 32-bit processor (or even a 64-bit processor) I doubt our techniques would look good. In Hyperscan, my regex project, we used SWAR for simple things (character scans) but I doubt that simdjson would work well if you tried to make it into swarjson. :-)

Post reply on HN