Any chance of something similar for CSV? (full RFC-4180 including quotes, escaping etc). Terabytes of "big data" get passed around as CSV.
CSV is on our list; this is a simpler task than JSON due to the absence of arbitrary nesting.
Simdjson – Parsing Gigabytes of JSON per Second
81–90 of 202 posts
Re: Simdjson – Parsing Gigabytes of JSON per Second
#82If you're working with json objects with sizes on the higher end quite often you're not going to need the entirety of them, just a small part of them. If that is the workload what then to do is simply parse as little data as possible: skip the validation, locate the relevant bits, and then start parsing, validation and all the stuff. In this optimizing the json scanner/lexer gives much greater improvement than optimi…
I agree that's a good strategy for big JSON. Do you know of any such "lazy" parsers? I think the problem is that to extract arbitrary keys, you really need to parse the whole thing, although you don't need to materialize nodes for the whole thing. But if you have big JSON with a given schema, you may be able to skip things lexically. You basically need to count {} and [], while taking into account " and \ within quot…
It’s not exactly the lazy parser you describe, but Sparser[1] builds filters to exclude json lines/files that can’t contain what you’re looking for, and only parses those that might.
The Morning Paper’s writeup[2] from last year provides a good summary
[1]: http://www.vldb.org/pvldb/vol11/p1576-palkar.pdf [2]: https://blog.acolyer.org/2018/08/20/filter-before-you-parse-...
Re: Simdjson – Parsing Gigabytes of JSON per Second
#83Earlier quoted context omitted.
https://en.wikipedia.org/wiki/JSON#Data_portability_issues : > Although Douglas Crockford originally asserted that JSON is a strict subset of JavaScript, his specification actually allows valid JSON documents that are invalid JavaScript. Specifically, JSON allows the Unicode line terminators U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR to appear unescaped in quoted strings, while ECMAScript 2018 and older doe…
That bit of incompatibility will be going away when this proposal is implemented, however: https://github.com/tc39/proposal-json-superset
It is already implemented in the current Firefox, Chrome and Safari 12.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#84> We store strings as NULL terminated C strings. Thus we implicitly assume that you do not include a NULL character within your string, which is allowed technically speaking if you escape it (\u0000).
I lost count to broken JSON parsers which all fall to that.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#85One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.
Any plan for wrapper for android?
Re: Simdjson – Parsing Gigabytes of JSON per Second
#86One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.
Any plan for wrapper for android?
This would imply an ARM port, I guess, as x86 android isn't much of a thing anymore AFAIK.
I don't think either of us know much about android - not enough to do that. But an ARM port is very interesting.
Since I'm no longer an Intel employee I don't see why I shouldn't skill up and do a Neon port (I got interested in SVE, but since ARM doesn't seem to want to bother releasing cores that run SVE, I'm not going to go too far down that path right now). Neon, on the other hand, is in tons of places. As far as I know all the required permutes, carryless multiplies and various other SIMD bits and pieces are there on Neon. So it's a simple matter of porting.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#87Earlier quoted context omitted.
From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…
... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.
[deleted]
Re: Simdjson – Parsing Gigabytes of JSON per Second
#88This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102
Yeah, Swift-most-everything is pretty slow, but particularly parsing/generating. Pre-Swift Foundation serialisation code was already...majestic, and in the Swift conversion they've typically managed to slow things down even further. Which didn't seem possible, but they managed.
I have given a bunch of talks[1] on this topic, there's also a chapter in my iOS/macOS performance book[2], which I really recommend if you want to understand this particular topic. I did really fast XML[3][4], CSV[5] and binary plist parsers[6] for Cocoa and also a fast JSON serialiser[7]. All of these are usually around an order of magnitude faster than their Apple equivalents.
Sadly, I haven't gotten around to doing a JSON parser. One reason for this is that parsing the JSON at character level is actually the smaller problem, performance-wise, same as for XML. Performance tends to be largely determined by what you create as a result. If you crate generic Foundation/Swift dictionaries/arrays/etc. you have already lost. The overhead of these generic data structure completely overwhelms the cost of scanning a few bytes.
So you need something more akin to a steaming interface, and if you create objects you must create them directly, without generic temporary objects. This is where XML is easier, because it has an opening tag that you can use to determine what object to create. With JSON, you get "{" so basically you have to know what structure level corresponds to what objects.
Maybe I should write that parser...
[1] https://www.google.com/search?hl=en&q=marcel%20weiher%20perf...
[2] https://www.amazon.com/gp/product/0321842847/
[3] https://github.com/mpw/Objective-XML
[4] https://blog.metaobject.com/2010/05/xml-performance-revisite...
[5] https://github.com/mpw/MPWFoundation/blob/master/Collections...
[6] https://github.com/mpw/MPWFoundation/blob/master/Collections...
[7] https://github.com/mpw/MPWFoundation/blob/master/Streams.sub...
Re: Simdjson – Parsing Gigabytes of JSON per Second
#89Earlier quoted context omitted.
... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.
You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…
> "It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats."
What.
typedef struct _some_struct_t
{
unsigned long some_long;
unsigned long some_other_long;
} some_struct_t;
...
{
some_struct_t foo = { 0 };
foo.some_long = 1;
foo.some_other_long = 2;
}
Re: Simdjson – Parsing Gigabytes of JSON per Second
#90> We store strings as NULL terminated C strings. Thus we implicitly assume that you do not include a NULL character within your string, which is allowed technically speaking if you escape it (\u0000). I lost count to broken JSON parsers which all fall to that.
Yeah, this is unforgivable, and for me makes the whole speed argument void.
Edit: to be fair, they handle a couple of other things, which many similar libraries ignore. I particulary like the support for full 64bit integers. And at least they document their limitation on NULL bytes.