Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

91–100 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#91

Post by 0db532a0 »

Jerry2 wrote: post #45
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

Why not use a timeseries database, like http://btrdb.io?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#92

Post by paulfurtado »

orbifold wrote: post #71
lifthrasiir wrote: post #69

Earlier quoted context omitted.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

This is a super flawed argument. Clearly flat buffers and even protocol buffers are faster to serialize and deserialize than json, regardless of what you benchmark in python.

Python might be the one language that isn't true for. In my Python experience, the Google protobuf library is frustratingly slower than the built-in json module for any data structures I've cared about, which is why things like pyrobuf exist to solve that performance problem: https://github.com/appnexus/pyrobuf

Re: Simdjson – Parsing Gigabytes of JSON per Second

#93

Post by lifthrasiir »

uasm wrote: post #89
lifthrasiir wrote: post #69

Earlier quoted context omitted.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

> "It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats." What. typedef struct _some_struct_t { unsigned long some_long; unsigned long some_other_long; } some_struct_t; ... { some_struct_t foo = { 0 }; foo.some_long = 1; foo.some_other_long = 2; } Is somehow comparable to using JSON?

C is one of extreme cases; that's why Cap'n'proto works pretty well in C++ and its cousins for example (it amortizes the decoding cost to accessors, and accessors are really cheap in those languages). There are many languages and implementations where decoding cost is not as significant.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#94

Post by [deleted] »

avodonosov wrote: post #74
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

Any chance to have a similar thing for s-expressions? I parse GBs of them and Common Lisp reader is very slow.

[deleted]

Re: Simdjson – Parsing Gigabytes of JSON per Second

#95

Post by Cthulhu_ »

saagarjha wrote: post #40
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

I ran one of the Codable benchmarks in instruments, and here's what the top functions were: 19.98 s swift_getGenericMetadata 19.15 s newJSONString 16.17 s objc_msgSend 15.33 s _swift_release_(swift::HeapObject*) 14.45 s tiny_malloc_should_clear 12.81 s _swift_retain_(swift::HeapObject*) 11.28 s searchInConformanceCache(swift::TargetMetadata const*, swift::TargetProtocolDescriptor const*) 10.46 s swift_dynamicCastImpl…

Can you see any differences with different levels of optimization? I recall a presentation at some point where the old obj-C style compiled code did a lot of checks before and after calling a method ("does this object listen to this message?"), while with an optimization option enabled (whole module optimization?) these calls could be optimized out. That is, with Swift they can make the resulting machine code less er, "checking for safety", so to speak.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#96

Post by mpweiher »

marton78 wrote: post #49
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

While theoretically true, in practice the actual character parsing tends to a small to negligible part of the overall time. Which leads to the measurable fact that on macOS/iOS, the JSON serialization stuff is actually one of their fastest, faster than their binary stuff.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#97

Post by Cthulhu_ »

teej wrote: post #73
Jerry2 wrote: post #45

Earlier quoted context omitted.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

You’ve kinda just described AWS Athena.

This comment needs to be higher up; Amazon has a service for doing just this, dumping 'dumb' files (like json, csv, etc) into S3 buckets and performing SQL queries on them. No need to have to think about how to store things for future querying.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#98

Post by IshKebab »

rapsey wrote: post #75
orbifold wrote: post #71

Earlier quoted context omitted.

This is a super flawed argument. Clearly flat buffers and even protocol buffers are faster to serialize and deserialize than json, regardless of what you benchmark in python.

And for the amount of messages that are being sent, the speed difference is irrelevant. This is the same conclusion sqlite developers came to. They tested turning JSON column types to binary and the speed difference was not large enough to warrant maintaining that code so they kept the data in JSON.

If the speed different is irrelevant, why are they struggling with it?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#99

Post by IshKebab »

lifthrasiir wrote: post #69
saurik wrote: post #58

Earlier quoted context omitted.

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

Are you using the slowness of Python's `struct` module to prove that binary formats in fast languages are slow?

I've benchmarked Capnp Vs JSON for Modern C++ in C++, and Capnp was something like 8 times faster.

If you're struggling with JSON performance how is moving to a binary format like Capnp (or Flatbuffers etc.) not a better solution?

Post reply on HN