Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

61–70 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#61

Post by uasm »

saurik wrote: post #58
aratno wrote: post #53

Earlier quoted context omitted.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

But... how else are the people that have never seen a byte array or had to flip endianness will be able to write plugins for my text editor?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#62

Post by groestl »

dlbucci wrote: post #7
avmich wrote: post #6

> All JSON is JavaScript, but not all JavaScript is JSON Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).

Basically saying any valid-format JSON is valid JS as well. But JSON doesn't have any programming features (or the nice things like non-quoted keys/trailing commas)

This is a dangerous assumption to make, and one that bit us a while ago when using trigger.io for an app.

We had a lot of user supplied data in the strings of our API responses, some of it copied from Word documents and were ridden with U+2028 and U+2029 whitespace. Turns out that on iOS, the trigger.io library makes the all too popular assumption that any well-formated JSON can be interpreted as JS, and parses the responses with "eval", thus turning all those unicode characters _within JSON strings_ into newlines!

Re: Simdjson – Parsing Gigabytes of JSON per Second

#63

Post by glangdale »

captbaritone wrote: post #56
oh_sigh wrote: post #46

Earlier quoted context omitted.

What if you're ingesting thousands or millions of small feeds? You might not have much control or desire to dictate format to your clients

The readme specifies that it’s not optimized for reading a large number of small files.

This would be an easy extension if you wanted to concatenate the files. The plumbing and API aren't there right now, but it isn't hard to see how to do it.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#64

Post by daveevad »

eesmith wrote: post #14
chubot wrote: post #11

Earlier quoted context omitted.

The JSON spec [1] never had any updates, so it couldn't have diverged. Kudos to Douglas Crockford for keeping it simple. I wish more standards committees would take a cue from him. (Looking at ECMAScript [2] and C++.) There's been a tremendous amount of growth and value around JSON precisely because it's so simple and easy to implement. People complain about the lack of comments and trailing commas, but I think those…

https://en.wikipedia.org/wiki/JSON#Data_portability_issues : > Although Douglas Crockford originally asserted that JSON is a strict subset of JavaScript, his specification actually allows valid JSON documents that are invalid JavaScript. Specifically, JSON allows the Unicode line terminators U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR to appear unescaped in quoted strings, while ECMAScript 2018 and older doe…

> JSON allows the Unicode line terminators U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR to appear unescaped in quoted strings, while ECMAScript 2018 and older does not.

My code has parsed a lot JSON and that is new data to me. Thank you for that!

Do you know the historical reasoning for this particular deviation? Are there any infamous bugs or common use cases this departure impacts?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#65

Post by glangdale »

defen wrote: post #59
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

How hard would it be to extend the parser to handle arbitrary-precision numbers? Strictly speaking the JSON spec does not require numbers to fit into 64-bit ints / doubles.

Daniel Lemire did most of the work on the number handling, but our general approach was to try to do work that's similar to what the bulk of other libraries do. I believe pretty much everyone throws oversize numbers on the floor.

I don't think it would be hard at all; it would just be extra effort that wasn't needed to run obvious comparisons.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#66

Post by PhilippGille »

Jerry2 wrote: post #45
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

If you are considering using plain files instead of a DB server, you could try a compromise and use an embedded key-value store like RocksDB, LevelDB, BadgerDB etc.

It's local storage only, limited query capabilities depending on the DB, but should be extremely fast.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#67

Post by e98cuenc »

saurik wrote: post #58
aratno wrote: post #53

Earlier quoted context omitted.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

It seems they're getting parsing times 1,000x slower than any other parser, 10,000x slower than simdjson. The complaint is understandable, but ironic :)

Re: Simdjson – Parsing Gigabytes of JSON per Second

#68

Post by nothrabannosir »

aratno wrote: post #53
marton78 wrote: post #49

Earlier quoted context omitted.

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

So is it too slow or not?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#69

Post by lifthrasiir »

saurik wrote: post #58
aratno wrote: post #53

Earlier quoted context omitted.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats.

As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` module) and simple binary formats (that would be an ideal case for Python `struct` module). Decoding the same simple value 1,000,000 times in CPython 3.6.4 took...

    Format  Size  Iters.     Speed
    ------  ----  ---------  ----------
    JSON      28    205,000   5.75 MB/s
    Struct     6  2,400,000  14.4  MB/s
Of course YMMV, but even the `struct` module was only 2--12 times (depending on what you care about) faster than the `json` module in this particular case. And this is really minimal, you need an (slow) interpreted code for more complex binary formats. Right, you can use PyPy for the JIT compilation or binary modules for sidestepping the interpreter overhead! The point is that, it of course matters, but not quite drastic improvements you'd imagine.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#70

Post by samsaga2 »

chairmanwow wrote: post #34
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

Any technical blog articles you have that explain how you were able to ascertain these incredible performance gains? Kudos on some incredible work! :)

Jsmn it's already pretty fast and simple. How the hell can be a lot faster than that? I'm very curious.
Post reply on HN