Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

121–130 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#121

Post by dkersten »

glangdale wrote: post #41
SoSKatan wrote: post #36

Earlier quoted context omitted.

Not really a question, but if you ever get to the point of wondering what a good next challenging project would be, consider generalizing some of these techniques into a next generation Yacc / Bison replacement. Something that can take generic grammer rules and turn it into a high performance parsing engine. It wouldn't have to support every possible grammar or option. Json isn't that complex of a language, but even…

It's on the list as a research project. It's not obvious to me at this stage that the bottlenecks for more advanced parsers are necessarily going to be in the same place as they are for JSON. It might make more sense to look at a state-of-the-art parser and see if we can contribute a few tricks instead.

That sounds interesting. Where is the best place to follow your future work? Your & Daniel Lemire's Github, or elsewhere?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#122

Post by saagarjha »

Cthulhu_ wrote: post #95
saagarjha wrote: post #40

Earlier quoted context omitted.

I ran one of the Codable benchmarks in instruments, and here's what the top functions were: 19.98 s swift_getGenericMetadata 19.15 s newJSONString 16.17 s objc_msgSend 15.33 s _swift_release_(swift::HeapObject*) 14.45 s tiny_malloc_should_clear 12.81 s _swift_retain_(swift::HeapObject*) 11.28 s searchInConformanceCache(swift::TargetMetadata const*, swift::TargetProtocolDescriptor const*) 10.46 s swift_dynamicCastImpl…

Can you see any differences with different levels of optimization? I recall a presentation at some point where the old obj-C style compiled code did a lot of checks before and after calling a method ("does this object listen to this message?"), while with an optimization option enabled (whole module optimization?) these calls could be optimized out. That is, with Swift they can make the resulting machine code less er…

This was done at -O I believe (whatever the default is for "Profiling" in Xcode). This is anecdotal, but the fact that the code isn't littered with _swift_retain/_swift_release calls probably means that most of the standard reference-counting boilerplate has been optimized away.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#123

Post by saagarjha »

raphlinus wrote: post #43
saagarjha wrote: post #40

Earlier quoted context omitted.

I ran one of the Codable benchmarks in instruments, and here's what the top functions were: 19.98 s swift_getGenericMetadata 19.15 s newJSONString 16.17 s objc_msgSend 15.33 s _swift_release_(swift::HeapObject*) 14.45 s tiny_malloc_should_clear 12.81 s _swift_retain_(swift::HeapObject*) 11.28 s searchInConformanceCache(swift::TargetMetadata const*, swift::TargetProtocolDescriptor const*) 10.46 s swift_dynamicCastImpl…

Yeah, I've done some analysis, it's creating a ton of objects to conform to the Codable protocol, and a lot of those objects are for codingPath, which is updated for basically every node in the tree. It's not a mystery, we just don't know the best way to fix it.

Is there a reason you need to use Codable? Sorry if this sounds uninformed, I haven't taken that much time to look at what you're doing exactly (I just ran https://github.com/jeremywiebe/json-performance).

Re: Simdjson – Parsing Gigabytes of JSON per Second

#124

Post by jillesvangurp »

Number handling looks like it would be a problem. There are Test suites for json parsers and lots of parsers that fail a lot of these tests. Check e.g. https://github.com/nst/JSONTestSuite which checks compliance against RFC 8259.

Publishing results against this could be useful both for assessing how good this parser is and establishing and documenting any known issues. If correctness is not a goal, this can still be fine but finding out your parser of choice doesn't handle common json emitted by other systems can be annoying.

Regarding the numbers, I've run into a few cases where Jackson being able to parse BigIntegers and BigDecimals was very useful to me. Silently rounding to doubles or floats can be lossy and failing on some documents just because the value exceeds max long/in t can be an issue as well.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#125

Post by dkersten »

dmix wrote: post #51
oh_sigh wrote: post #46

Earlier quoted context omitted.

What if you're ingesting thousands or millions of small feeds? You might not have much control or desire to dictate format to your clients

Yeah not everyone, I’d even say the majority of people, are using software parsing libraries where they are in control of the input data format.

For storing stuff yourself, sure, but as a web developer, most data I consume is JSON served by some third-party REST API and the format they serve me is definitely not under my control. Anecdotally, most developers I know or have spoken to are in similar situations for a large portion of their data-processing needs (at least, for stuff that's not in a database, although even in DB's, JSON is increasingly popular for a number of reasons).

Even for output, there is the common case where your clients expect JSON because its the de facto standard and is super accessible (every language has parsers for it), so you have little choice but to serve your data as JSON.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#126

Post by carlmr »

imtringued wrote: post #110
glangdale wrote: post #81

Earlier quoted context omitted.

CSV is on our list; this is a simpler task than JSON due to the absence of arbitrary nesting.

I doubt someone using CSV for big data is going to follow that rule...

What do you mean? It's not a rule, it's just not possible in the CSV format to have arbitrary nesting.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#127

Post by glangdale »

dkersten wrote: post #121
glangdale wrote: post #41

Earlier quoted context omitted.

It's on the list as a research project. It's not obvious to me at this stage that the bottlenecks for more advanced parsers are necessarily going to be in the same place as they are for JSON. It might make more sense to look at a state-of-the-art parser and see if we can contribute a few tricks instead.

That sounds interesting. Where is the best place to follow your future work? Your & Daniel Lemire's Github, or elsewhere?

I might go so far as to post to branchfree.org, and Daniel posts at https://lemire.me/blog/ so either of those, plus github, ought to cover it.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#128

Post by philbo »

chubot wrote: post #13
xfs wrote: post #10

If you're working with json objects with sizes on the higher end quite often you're not going to need the entirety of them, just a small part of them. If that is the workload what then to do is simply parse as little data as possible: skip the validation, locate the relevant bits, and then start parsing, validation and all the stuff. In this optimizing the json scanner/lexer gives much greater improvement than optimi…

I agree that's a good strategy for big JSON. Do you know of any such "lazy" parsers? I think the problem is that to extract arbitrary keys, you really need to parse the whole thing, although you don't need to materialize nodes for the whole thing. But if you have big JSON with a given schema, you may be able to skip things lexically. You basically need to count {} and [], while taking into account " and \ within quot…

For node.js, I wrote a lib that can selectively parse JSON subtrees:

https://gitlab.com/philbooth/bfj

The specific function of interest here is `bfj.match`, which takes a readable stream and a selector as arguments:

https://gitlab.com/philbooth/bfj#how-do-i-selectively-parse-...

It still walks the full tree like a regular parser, but just avoids creating any data items unless the selector matches. Though there is an outstanding issue to support JSONPath in the selector, currently it only matches individual keys and values.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#129

Post by hrdwdmrbl »

sbr464 wrote: post #108
hrdwdmrbl wrote: post #100

Would it be possible to make a native module out of this for node?

Here's the node bindings for rapid json, I'm assuming it would be similar. https://github.com/matthewpalmer/node-rapidjson

Thank you!

Though from the readme on that module the dev says "it turns out that you’re better off using the normal Node.js/V8 implementation unless you’re operating on huge JSON.

... the bridging from V8 to C++ is a bit too costly at this stage."

Re: Simdjson – Parsing Gigabytes of JSON per Second

#130

Post by uasm »

osmarks wrote: post #114
uasm wrote: post #106

Earlier quoted context omitted.

> "C is one of extreme cases" I would say it's the other way around. We've had the knowledge and tools to build performant, scalable and highly maintainable systems for a while now. The learning curve is there, but that's part of the trade. We've been too occupied with reducing the entry barrier though - the end result being people shoving JSON into places it should have never been in. JSON can absolutely be a part o…

Because the idea of Xi is that it can support different frontends for different platforms, and that probably wouldn't work out to well if they all had to be in C.

The Xi backend is already written in Rust, a relatively low-level language with a somewhat C-like FFI/ABI. The choice to use JSON in time-critical code, when more performant alternatives are available, seems to me like a mistake.
Post reply on HN