Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

41–50 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#41

Post by glangdale »

SoSKatan wrote: post #36
SoSKatan wrote: post #28

Earlier quoted context omitted.

I've written my fare share of performant code over the years, but this is some next level shit. I've been reading it the last few hours. The only question I have is what is the term for that place considered two degrees past black magic? Since you live there, I have to assume you know the name.

Not really a question, but if you ever get to the point of wondering what a good next challenging project would be, consider generalizing some of these techniques into a next generation Yacc / Bison replacement. Something that can take generic grammer rules and turn it into a high performance parsing engine. It wouldn't have to support every possible grammar or option. Json isn't that complex of a language, but even…

It's on the list as a research project. It's not obvious to me at this stage that the bottlenecks for more advanced parsers are necessarily going to be in the same place as they are for JSON. It might make more sense to look at a state-of-the-art parser and see if we can contribute a few tricks instead.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#43

Post by raphlinus »

saagarjha wrote: post #40
raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

I ran one of the Codable benchmarks in instruments, and here's what the top functions were: 19.98 s swift_getGenericMetadata 19.15 s newJSONString 16.17 s objc_msgSend 15.33 s _swift_release_(swift::HeapObject*) 14.45 s tiny_malloc_should_clear 12.81 s _swift_retain_(swift::HeapObject*) 11.28 s searchInConformanceCache(swift::TargetMetadata const*, swift::TargetProtocolDescriptor const*) 10.46 s swift_dynamicCastImpl…

Yeah, I've done some analysis, it's creating a ton of objects to conform to the Codable protocol, and a lot of those objects are for codingPath, which is updated for basically every node in the tree. It's not a mystery, we just don't know the best way to fix it.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#45

Post by Jerry2 »

glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status reports per day, that's going to be a crapton on files... but that might be faster to process than querying the database.

If you or anyone else has some opinions on this, please let me know! I'd really like to learn how people do this type of analysis at scale.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#46

Post by oh_sigh »

adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

What if you're ingesting thousands or millions of small feeds? You might not have much control or desire to dictate format to your clients

Re: Simdjson – Parsing Gigabytes of JSON per Second

#47

Post by Ultimatt »

scottlamb wrote: post #29
aristidb wrote: post #15

Earlier quoted context omitted.

Also noteworthy that on Intel at least, using AVX/AVX2 reduces the frequency of the CPU for a while. It can even go below base clock.

iirc, it's complicated. Some instructions don't reduce the frequency; some reduce it a little; some reduce it a lot. I'm not sure AVX2 is as ubiquitous as the README says: "We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel." I guess "mainstream" is somewhat subjective, but some recent Chromebooks have Celeron processors with no AVX2: https://us-store.acer.com/…

Because someone wanting 2.2GB/s JSON parsing is deploying to a chromebook...

Re: Simdjson – Parsing Gigabytes of JSON per Second

#49

Post by marton78 »

raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#50

Post by groestl »

adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

Maybe they want to convert incoming JSON to a binary serialization format to save bandwith, storage and CPU time on the rest of the pipeline ;)
Post reply on HN