101–110 of 202 posts
Earlier quoted context omitted.
And for the amount of messages that are being sent, the speed difference is irrelevant. This is the same conclusion sqlite developers came to. They tested turning JSON column types to binary and the speed difference was not large enough to warrant maintaining that code so they kept the data in JSON.
If the speed different is irrelevant, why are they struggling with it?
I guess the question is, what do you parse it to? I'm guessing definitely not turning objects into std::unordered_map and arrays into std::vector or some such. So how easy it is to use the "parsed" data structure? How easy is it to add an element to some deeply nested array for example?
The ParsedJson type is immutable and accessed mutating iterators (up and down the tree, forward and backward through members and indices). My immediate thought is to compare it to rapidjson, which I've used before. The paradigm of mutating iterators seems awkward at first but should be just as powerful as rapidjson's Value. For example, both approaches end up doing a linear scan to find an object member by name. The…
Earlier quoted context omitted.
The ParsedJson type is immutable and accessed mutating iterators (up and down the tree, forward and backward through members and indices). My immediate thought is to compare it to rapidjson, which I've used before. The paradigm of mutating iterators seems awkward at first but should be just as powerful as rapidjson's Value. For example, both approaches end up doing a linear scan to find an object member by name. The…
Is there a reason these fast json libraries seem to favor doing linear scan for object representation?
JSON Objects tend to have few enough values that it doesn't matter a ton anyway.
One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.
By the way, nativejson-benchmark (from RapidJson) has a nice conformance checker that tries various corner cases. But you probably know it.
Earlier quoted context omitted.
> "It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats." What. typedef struct _some_struct_t { unsigned long some_long; unsigned long some_other_long; } some_struct_t; ... { some_struct_t foo = { 0 }; foo.some_long = 1; foo.some_other_long = 2; } Is somehow comparable to using JSON?
C is one of extreme cases; that's why Cap'n'proto works pretty well in C++ and its cousins for example (it amortizes the decoding cost to accessors, and accessors are really cheap in those languages). There are many languages and implementations where decoding cost is not as significant.
I would say it's the other way around.
We've had the knowledge and tools to build performant, scalable and highly maintainable systems for a while now. The learning curve is there, but that's part of the trade. We've been too occupied with reducing the entry barrier though - the end result being people shoving JSON into places it should have never been in.
JSON can absolutely be a part of a text editor's architecture - with areas that don't necessarily require near real time performance (think configuration, metrics). Anything beyond that - C structs would be a great way to go, and I don't see why there's a debate here.
One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.
The big difference between RapidJson and sajson is surprising to me. When I benchmarked them their performance was comparable: https://github.com/project-gemmi/benchmarking-json . Did you use RapidJson in full-precision mode? By the way, nativejson-benchmark (from RapidJson) has a nice conformance checker that tries various corner cases. But you probably know it.
We use RapidJSON in the high-performance mode not the funky mode that minimizes FP error (which is some astounding work - I had no idea that strtof was so involved!). Number conversion is not our #1 focus - doing it well is nice, but all implementations have access to the same FP tricks, so you don't really learn much by going wild on this aspect.
At least, you don't unless FP conversion is your focus, in which case you should share your FP conversion code with everyone!
Would it be possible to make a native module out of this for node?
> We store strings as NULL terminated C strings. Thus we implicitly assume that you do not include a NULL character within your string, which is allowed technically speaking if you escape it (\u0000). I lost count to broken JSON parsers which all fall to that.
Yeah, this is unforgivable, and for me makes the whole speed argument void. Edit: to be fair, they handle a couple of other things, which many similar libraries ignore. I particulary like the support for full 64bit integers. And at least they document their limitation on NULL bytes.
Please add an issue on Github.
Edit: I went ahead and added an issue. Seems like something we should fix.
Any chance of something similar for CSV? (full RFC-4180 including quotes, escaping etc). Terabytes of "big data" get passed around as CSV.
CSV is on our list; this is a simpler task than JSON due to the absence of arbitrary nesting.