Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

71–80 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#71

Post by orbifold »

lifthrasiir wrote: post #69
saurik wrote: post #58

Earlier quoted context omitted.

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

This is a super flawed argument. Clearly flat buffers and even protocol buffers are faster to serialize and deserialize than json, regardless of what you benchmark in python.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#72

Post by teej »

adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

That’s a nice sentiment but we don’t always get to choose.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#73

Post by teej »

Jerry2 wrote: post #45
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

You’ve kinda just described AWS Athena.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#74

Post by avodonosov »

glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

Any chance to have a similar thing for s-expressions? I parse GBs of them and Common Lisp reader is very slow.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#75

Post by rapsey »

orbifold wrote: post #71
lifthrasiir wrote: post #69

Earlier quoted context omitted.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

This is a super flawed argument. Clearly flat buffers and even protocol buffers are faster to serialize and deserialize than json, regardless of what you benchmark in python.

And for the amount of messages that are being sent, the speed difference is irrelevant.

This is the same conclusion sqlite developers came to. They tested turning JSON column types to binary and the speed difference was not large enough to warrant maintaining that code so they kept the data in JSON.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#76

Post by idunno246 »

Jerry2 wrote: post #45
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

reading lots of small files on s3 or local filesystems is tricky. a million devices with one dozen files, so lets say 12 million files.

One thing locally is each file takes up a full block. So even if you only need 500 bytes of data in a file, and a block is 4kb, youve wasted 3.5kb of space and IO. Multiply that by a million and youre wasting gigabytes of space.

In S3, listing 12 million files takes 12 thousand http(max return is 1000 items). So that would take two minutes if you assume its 10ms per round trip. Let's say you wanted to read each file, and again each read takes 10ms.. youre looking at 1.4 days. Obviously this can be parallelized, but when you look at the raw byte size this is a huge overhead, and this is just to read one day of data.

If you concatenate the files together to get a reasonable size and number of files, raw json on s3 is really powerful. Point athena at it, and you just write sql and it handles the rest, and is serverless. But it does make single row lookups more expensive(supplementing with dynamodb could keep it serverless if single row lookups are frequent).

lots of optimizations will get improvements, like parquet that tobilg mentioned(binary format and columnar), but anything with a decent file size will work.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#78

Post by lifthrasiir »

orbifold wrote: post #71
lifthrasiir wrote: post #69

Earlier quoted context omitted.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

This is a super flawed argument. Clearly flat buffers and even protocol buffers are faster to serialize and deserialize than json, regardless of what you benchmark in python.

So you claim that decoding flat buffers and protobuf is faster than decoding with `struct`? I'm pretty much aware of various flaws and even stated some, but I barely buy that claim without a separate benchmark (which I really welcome by the way).

At least I fully understand what the `struct` module actually does under the hood---it sorta compiles to a list of fields and "interprets" the dead simple VM in C. Oh, of course I've used the precompiled `struct.Struct` for that reason (but it was only 20% faster). Anyway, this arrangement is typical for most schematic serialization formats in any language: a bunch of function calls for gluing the desired format, plus a set of well-optimized core functions (not necessarily written in C :-). Henceforth my justification that this is close to the "bare-bones" serialization format.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#79

Post by z3t4 »

adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

I agree. But JSON serialization is very complicated for very little gain. It would make it impossible to do things like opening the json file in an editor to change some property names. So watch out for premature optimization.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#80

Post by glangdale »

avodonosov wrote: post #74
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

Any chance to have a similar thing for s-expressions? I parse GBs of them and Common Lisp reader is very slow.

Probably not too hard. It would come down to how easy it is to detect quoting conventions so you don't accidentally parse () chars in strings. JSON is medium-easy. I don't know where the canonical definition of s-expressions you're using comes from (is it just Common Lisp?) so I don't know how this works.

We'd like to have some more examples of formats people care about - I'm interested in generalizing this work. So if you want to followup with more detail please do.

Post reply on HN