Simdjson – Parsing Gigabytes of JSON per Second • Hacker News Board

Re: Simdjson – Parsing Gigabytes of JSON per Second

#151

Post by jchb » Thu, Feb 21, 2019, 3:14 PM UTC

raphlinus wrote: post #148

nothrabannosir wrote: post #68

Earlier quoted context omitted.

So is it too slow or not?

We actually do get 60fps, but JSON parsing on the Swift side takes more than its share of total CPU load, affecting power consumption among other things. So (partly to address the trolls elsewhere in the thread), the choice of JSON does not preclude fast implementation (as the existence of simdjson proves), but it does make it dependent on the language having a performant JSON implementation. I made the assumption th…

One of the other performant libraries in the comparison section of simdjson has a Swift wrapper: https://github.com/chadaustin/sajson. Haven't tried it, but one option would be to bring that up to date. Another option, now that Swift 5 strings use utf8 as a native encoding, it may be possible to write a fast Json parser in native Swift. Likely someone already has or is doing that.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#152

Post by andonisus » Thu, Feb 21, 2019, 3:19 PM UTC

Jerry2 wrote: post #45

glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

I am replying to you as an engineer at an IoT company that provides SaaS in AWS for the data our devices produce. To solve this problem, we transmit our data in a proprietary "raw" binary format that then gets parsed into a protobuf. All data for a given UTC day is appended to this protobuf file and hosted in S3. Retrieving data requires downloading the protobuf file from S3, unmarshalling the protobuf, and finding the entry you care about.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#153

Post by sbr464 » Thu, Feb 21, 2019, 3:19 PM UTC

hrdwdmrbl wrote: post #129

sbr464 wrote: post #108

Earlier quoted context omitted.

Here's the node bindings for rapid json, I'm assuming it would be similar. https://github.com/matthewpalmer/node-rapidjson

Thank you! Though from the readme on that module the dev says "it turns out that you’re better off using the normal Node.js/V8 implementation unless you’re operating on huge JSON. ... the bridging from V8 to C++ is a bit too costly at this stage."

That was two years ago though, not sure what improvements the N-API has in newer versions of nodejs.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#154

Post by DougBTX » Thu, Feb 21, 2019, 3:33 PM UTC

wongarsu wrote: post #119

adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

I have in the past parsed terabytes of JSON. The specific use case was analysing archived Reddit comments. The Reddit API uses JSON, and somebody [1] runs a server that just dumps them in a file, one line of JSON per comment, and offers them for download (compressed, obviously). So now you end up with Gigabytes of small JSONs per month, and anything you do will be quickly dominated by JSON parsing time. You could sto…

> either your binary format ends up not much better than JSON or you end up reencoding old comments because the API changed

Those are other options too, eg, storing the schema separately from the records (then batching records with identical schemas in compact binary files) and defining migration rules between different schemas (eg, if schema A has required field "foo" while schema B has required field "foo" and optional field "bar" then data which follows schema A can be trivially migrated to schema B at read time without needing to reencode on disk).

https://avro.apache.org/docs/current/

Re: Simdjson – Parsing Gigabytes of JSON per Second

#155

Post by dilap » Thu, Feb 21, 2019, 3:59 PM UTC

raphlinus wrote: post #148

nothrabannosir wrote: post #68

Earlier quoted context omitted.

So is it too slow or not?

We actually do get 60fps, but JSON parsing on the Swift side takes more than its share of total CPU load, affecting power consumption among other things. So (partly to address the trolls elsewhere in the thread), the choice of JSON does not preclude fast implementation (as the existence of simdjson proves), but it does make it dependent on the language having a performant JSON implementation. I made the assumption th…

At some point though, isn't it maybe easier just to use an inherently more efficient format than trying to rely on clever implementations to save you?

I totally get json for public internet services where you want to have lots of consumers and using a more efficient format would be significant friction, but writing an editor frontend is a very large endeavor -- it seems like the extra work of adopting something more efficient than json (like flatbuffers or whatever) would really be in the noise.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#156

Post by floofy222 » Thu, Feb 21, 2019, 4:17 PM UTC

glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I see that you are using MMX intrinsics directly, like _mm_sub_pi8, but you are never calling _mm_empty (https://software.intel.com/sites/landingpage/IntrinsicsGuide...) as required by the SysV AMD64 ABI (and pretty much all other ABIs out there).

I think the behavior of all the code that touches is undefined (it breaks the calling convention of the ABI), and while this often results in corrupted floating point values in registers, maybe you won't see much if you are not using the FPU. Still, since the function is inline, chances that this gets inlined somewhere where it could cause trouble seem high.

You might want to look into that.

Also, I wish this would all be written in Rust, there is great portable SIMD support over there. Might make your life easier trying to target other platforms.

EDIT: as burntsushi mentions below, that's not available in stable Rust, but if you want to squeeze out the last once of performance out of the Rust compiler, chances are you won't be using that anyways.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#157

Post by anderspitman » Thu, Feb 21, 2019, 4:22 PM UTC

aratno wrote: post #53

marton78 wrote: post #49

Earlier quoted context omitted.

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

I'm going off topic at this point but I'd think for a native app the main advantages of a binary format would be the static typing and code generation that come from using an IDL.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#158

Post by raphlinus » Thu, Feb 21, 2019, 4:33 PM UTC

dilap wrote: post #155

raphlinus wrote: post #148

Earlier quoted context omitted.

We actually do get 60fps, but JSON parsing on the Swift side takes more than its share of total CPU load, affecting power consumption among other things. So (partly to address the trolls elsewhere in the thread), the choice of JSON does not preclude fast implementation (as the existence of simdjson proves), but it does make it dependent on the language having a performant JSON implementation. I made the assumption th…

At some point though, isn't it maybe easier just to use an inherently more efficient format than trying to rely on clever implementations to save you? I totally get json for public internet services where you want to have lots of consumers and using a more efficient format would be significant friction, but writing an editor frontend is a very large endeavor -- it seems like the extra work of adopting something more…

It's a complicated tradeoff. It's not just performance, the main thing is clear code. Another factor was support across a wide variety of languages, which was thinner for things like flatbuffers at the time we adopted JSON. Also, "clever implementations" like simdjson don't have a high cost, if they're nice open source libraries.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#159

Post by owenmarshall » Thu, Feb 21, 2019, 4:34 PM UTC

teej wrote: post #73

Jerry2 wrote: post #45

Earlier quoted context omitted.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

You’ve kinda just described AWS Athena.

I've used Athena really effectively to solve similar problems. If your data storage is relatively small and/or your queries relatively infrequent, JSON can be a good fit. As one of those dimensions expands, you can decrease costs/increase performance by converting to Parquet and compressing.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#160

Post by twic » Thu, Feb 21, 2019, 4:35 PM UTC

Jerry2 wrote: post #45

glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…