What's the current state of the art in doing this on GPU?
To my knowledge, it is limited to posting "Towards JSON Parsing on a GPU" type articles. Writing that sort of article is easy and fun, without the tedious burden of implementing things.
Simdjson – Parsing Gigabytes of JSON per Second
181–190 of 202 posts
Re: Simdjson – Parsing Gigabytes of JSON per Second
#182This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102
What are you using? Have you tried NSJSONSerialization? It’s quite fast (am very curious how it shows in these benchmarks), but I don’t think it does the fancy Codable stuff.
You might want to check out the benchmark I wrote to compare exactly that.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#183Earlier quoted context omitted.
If they want to squeeze out the last ounce of performance out of the Rust toolchain it probably wouldn't make sense to use stable Rust anyways, so I don't think that's a big downside. Also, they are already relying on "unstable" (non-standard conforming) C++ features (e.g. the code uses non-standard attributes behind macros, etc.). Using nightly Rust isn't worse than that per se. Using Rust does have downsides. For t…
Sure. I'm just trying to be careful that we aren't going around advertising features that aren't stable yet without specifically saying that they aren't stable. It leads to a disappointing expectations mismatch. I do think stable Rust is perfectly capable though. I don't generally target nightly Rust and am happy with how much I can squeeze out of it. :-) (Check out the benchmarks for the memchr crate, which use SIMD…
Most programming languages don't have a stable / unstable distinction at all, and unless one is "in the Rust loop", stating something like "_unstable_ Rust can do X" won't probably mean what the reader think it means.
Unstable Rust sounds very dangerous, like something that breaks every day. Definitely more dangerous than stable Rust.
Yet if one is in the Rust loop, one knows that this is often not the case. I've been using some unstable features on nightly, like const fn, specialization, function traits, etc. for years (3 years?), and I've never had a CI build job fail due to a change to the implementation of these features.
Yet some features in stable Rust like Rust2018 uniform_paths or stable SIMD have caused many build job breaks and undefined behavior due to bugs in the compiler over the last months.
So whatever stability means, it does not mean "using this feature won't result in your code not breaking". It also doesn't mean "you have to use a nightly toolchain to use the feature".
An unstable Rust feature is more like a "compiler extension" in C / C++. It is just something that hasn't fully gone through the process of standardization.
I don't think it is a fair characterization that code that uses this extensions is not Rust. Pretty much all C++ code uses compiler extensions, and nobody says that this code is not C++ just because it uses one of them.
Explaining all of this when telling someone "Rust is a technology that allows you to solve problem X nicely" isn't helpful.
Many people vocal about Rust seem to think that Rust is the end in of itself. The goal isn't solving a problem, but using Rust to solve it. I see many of these people argue that unstable Rust isn't Rust, and that people should be using stable Rust etc. For most people, using Rust isn't the goal, solving their problem is. Whether one or many compiler extensions have to be enabled for that is pretty much irrelevant to them. Sure it would be nice if one didn't need to do that, but it isn't a big deal either. The big embedded community is living proof of that. Only a small minority of this community cares about the language enough to participate in its evolution. Most people don't care enough about that, they have more interesting problems to solve.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#184Earlier quoted context omitted.
Sure. I'm just trying to be careful that we aren't going around advertising features that aren't stable yet without specifically saying that they aren't stable. It leads to a disappointing expectations mismatch. I do think stable Rust is perfectly capable though. I don't generally target nightly Rust and am happy with how much I can squeeze out of it. :-) (Check out the benchmarks for the memchr crate, which use SIMD…
Most programming languages don't have a stable / unstable distinction at all, and unless one is "in the Rust loop", stating something like "_unstable_ Rust can do X" won't probably mean what the reader think it means. Unstable Rust sounds very dangerous, like something that breaks every day. Definitely more dangerous than stable Rust. Yet if one is in the Rust loop, one knows that this is often not the case. I've bee…
You happened to pick some features where there hasn’t been much development work, since other things were prioritized. And one feature that’s not mostly compiler internal.
This is not the general case for unstable features. And promoting the use of them too heavily can cause a lot of problems. It undermines trust in the language, especially given rust’s pre-1.0 reputation (which was well deserved at the time.)
Stuff that’s unstable isn’t in Rust; that’s why it can be changed or even wholesale removed at any time. The distinction is very important.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#185Will this work on JSON files that are larger than the available system memory? Firebase backups are huge JSON files and we haven’t found a good way to deal with them. There are some “streaming JSON parsers” that we have wrestled with but they are buggy.
Sadly it will not. Arguably we could 'stream' things, but we don't have an API or a use case for it. If you could capture your requirements and put them on an issue on Github, it would be helpful. We're not against the streaming use case, we just don't understand it very well.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#186One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.
Jackson benchmarks? I've heard it's twice as fast as rapidjson.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#187I guess the question is, what do you parse it to? I'm guessing definitely not turning objects into std::unordered_map and arrays into std::vector or some such. So how easy it is to use the "parsed" data structure? How easy is it to add an element to some deeply nested array for example?
into a token stream?
Re: Simdjson – Parsing Gigabytes of JSON per Second
#188I guess the question is, what do you parse it to? I'm guessing definitely not turning objects into std::unordered_map and arrays into std::vector or some such. So how easy it is to use the "parsed" data structure? How easy is it to add an element to some deeply nested array for example?
into a token stream?
Isn't that just lexing?
Re: Simdjson – Parsing Gigabytes of JSON per Second
#189> All JSON is JavaScript, but not all JavaScript is JSON Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).
The JSON spec [1] never had any updates, so it couldn't have diverged. Kudos to Douglas Crockford for keeping it simple. I wish more standards committees would take a cue from him. (Looking at ECMAScript [2] and C++.) There's been a tremendous amount of growth and value around JSON precisely because it's so simple and easy to implement. People complain about the lack of comments and trailing commas, but I think those…
> People complain about the lack of comments and trailing commas,
Yeah, I don't think JSON should include those things. I think the lack of comments makes JSON a poor format for config files, but that just means you should use another format for config files. JSON is good for machine-to-machine communication.
Re: Simdjson – Parsing Gigabytes of JSON per Second
#190> All JSON is JavaScript, but not all JavaScript is JSON Really? I thought they diverged specifications long enough ago (though using those extras could be discouraged in some cases).
The JSON spec [1] never had any updates, so it couldn't have diverged. Kudos to Douglas Crockford for keeping it simple. I wish more standards committees would take a cue from him. (Looking at ECMAScript [2] and C++.) There's been a tremendous amount of growth and value around JSON precisely because it's so simple and easy to implement. People complain about the lack of comments and trailing commas, but I think those…
Agree.
This is another useful resource, discussed here already - http://seriot.ch/parsing_json.php - which lists relevant standards. But "the" standard is static, so divergence, is any, is with other standards (different from json.org) vs. evolving JavaScript.