Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

191–200 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#191
post #45

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

well if you need indexed lookups, then use a database

if you're doing "table scan" processing of entire datasets, sure just-a-bunch-of-files would work too.

Databases can be surprisingly fast for things like that, since high performance file i/o is full of tricky/annoying stuff that databases have already optimized for.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#192

Earlier quoted context omitted.

Most programming languages don't have a stable / unstable distinction at all, and unless one is "in the Rust loop", stating something like "_unstable_ Rust can do X" won't probably mean what the reader think it means. Unstable Rust sounds very dangerous, like something that breaks every day. Definitely more dangerous than stable Rust. Yet if one is in the Rust loop, one knows that this is often not the case. I've bee…

You happened to pick some features where there hasn’t been much development work, since other things were prioritized. And one feature that’s not mostly compiler internal. This is not the general case for unstable features. And promoting the use of them too heavily can cause a lot of problems. It undermines trust in the language, especially given rust’s pre-1.0 reputation (which was well deserved at the time.) Stuff…

> Stuff that’s unstable isn’t in Rust; that’s why it can be changed or even wholesale removed at any time. The distinction is very important.

I've seen you talk about "writing an OS kernel in Rust", but never heard you phrase that as "writing an OS in kernel in _unstable_ Rust". I've never seen you stating: "correction: what you are using for embedded development, networking, etc. is not Rust, _but unstable Rust_" on any of the many blog posts, announcements, news, etc. about these topics over the past couple of years. I've seen you reply with that argument every now and then, when someone like me downplays the importance of the distinction, but I've never seen you address the source of that behavior.

If the distinction between Rust, and unstable Rust, is important. Why are the people at the top not making it? If you are working on the compiler, servo, etc. you are actually not programming in Rust, but in _unstable_ Rust all of the time. Are they hypocrites? I don't think so.

If I reflect on why I feel that this distinction is not important, the first thing I realize is that I do think the distinction is important. But this distinction is not binary _to me_, as opposed to how you and burntsushi are putting it.

As you mentioned, some unstable features change more than others. There is a wide range of how much continues breakage does using certain unstable features cause downstream users. Some features break every day, some features haven't broken anything in 3 years.

Are unstable features that haven't broken anything in 3 years stable? No, by definition, they aren't.

Are they practical to use? The answer isn't yes or no, the answer is "depends on how much breakage you are willing to accept". We upgrade C++ compiler ~twice per year, and even though we only write 100% standard compliant code, we have to always fix breakage due to the upgrade. Yet I wouldn't say that standard compliant C++ is an unstable programming language.

So, if consider bi-yearly breakage is stable enough for our professional C++ projects in practice, why would I judge Rust stable / unstable features using a different bar? This does not mean that I believe that using unstable (or only stable) features will never cause breakage, since that is impossible.

I've had stable Rust CI jobs break because the standard library added some new trait method, and that caused an ambiguity that broke in my stable Rust code. The answer was: your code was correct, but we are allowed to break it in this way.

In my opinion, it is not "stable vs unstable", but 99% vs what degree of stability does your project need, where choosing more stability than what it needs puts it at a technical disadvantage. It doesn't matter whether one is talking here about Rust unstable features, or using the super unstable next-gen stable Rust web framework.

The stability line does not lie where I or anybody else decides to arbitrarily put it. It lies exactly on the amount of stability that a particular project can tolerate, and it is up to the judgement of the developers of that particular project to find out where that is.

Telling someone that a particular project is not Rust because the stability line for that project does not fall where your line does feels just wrong. Particularly when those doing it don't make that distinctions about themselves and the projects their work on.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#193
post #53

Earlier quoted context omitted.

From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty…

I'm going off topic at this point but I'd think for a native app the main advantages of a binary format would be the static typing and code generation that come from using an IDL.

Rust (the language the Xi core is developed in) has static typing for JSON, as well as other serialization formats: https://github.com/serde-rs/json

Re: Simdjson – Parsing Gigabytes of JSON per Second

#194

Earlier quoted context omitted.

You happened to pick some features where there hasn’t been much development work, since other things were prioritized. And one feature that’s not mostly compiler internal. This is not the general case for unstable features. And promoting the use of them too heavily can cause a lot of problems. It undermines trust in the language, especially given rust’s pre-1.0 reputation (which was well deserved at the time.) Stuff…

> Stuff that’s unstable isn’t in Rust; that’s why it can be changed or even wholesale removed at any time. The distinction is very important. I've seen you talk about "writing an OS kernel in Rust", but never heard you phrase that as "writing an OS in kernel in _unstable_ Rust". I've never seen you stating: "correction: what you are using for embedded development, networking, etc. is not Rust, _but unstable Rust_" on…

You are making a mountain out of a molehill. We're on HN, not some Rust community space. Context is paramount. If saying, "portable SIMD is available on nightly Rust" or "portable SIMD is available as an experimental extension in Rust" feels better to you than "portable SIMD is available on unstable Rust," then go for it.

There's nothing binary about my position. My only point is to mitigate an expectation mismatch. People get pissed off when they're led to believe that a feature is baked and ready to use, when it actually isn't. Honestly, you've turned a simple correction into a ranty spiraling sub-thread. It's obnoxious.

You're also getting way too hung up on what stability means. "stability" in Rust, in the context of API availability, is a statement of intent and commitment, not a statement of how often a build will break. Of course, there may be a strong correlation between them!

Re: Simdjson – Parsing Gigabytes of JSON per Second

#195
post #45

Earlier quoted context omitted.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

reading lots of small files on s3 or local filesystems is tricky. a million devices with one dozen files, so lets say 12 million files. One thing locally is each file takes up a full block. So even if you only need 500 bytes of data in a file, and a block is 4kb, youve wasted 3.5kb of space and IO. Multiply that by a million and youre wasting gigabytes of space. In S3, listing 12 million files takes 12 thousand http(…

Yeah, this is what Kinesis Firehose is for. Send all of your messages there and it will batch them to S3.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#196
post #57
post #45

Earlier quoted context omitted.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

We‘re using AWS Kinesis delivery streams to batch incoming JSON messages from IoT devices to Parquet files in S3. Those can directly be read by different AWS services like Redshift, EMR or Athena...

What bothers me about Kinesis is that it is prohibitively expensive at scale if you don't compress your data before putting it to Kinesis.

But if you want to use the nice features like parquet conversion your data can't be compressed.

If it could handle compressed data at the same price I would use a lot more of it.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#197
post #135

Earlier quoted context omitted.

So is it too slow or not?

It's not a binary yes/no question. Given equally-high quality JSON and binary serdes, JSON is sufficiently fast. Raphlinus is saying that Swift's built-in deserialiser is obnoxiously slow.

Any reason not to just use a third party Swift JSON library?

Re: Simdjson – Parsing Gigabytes of JSON per Second

#198
post #55

Earlier quoted context omitted.

iirc, it's complicated. Some instructions don't reduce the frequency; some reduce it a little; some reduce it a lot. I'm not sure AVX2 is as ubiquitous as the README says: "We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel." I guess "mainstream" is somewhat subjective, but some recent Chromebooks have Celeron processors with no AVX2: https://us-store.acer.com/…

AVX2 also incurs some pretty large penalties for switching between SSE and AVX2. Depending on the amount of time taken in the library between calls, it could be problematic. This looks mostly applicable to server scenarios where the runtime environment is highly controlled.

There is no real penalty for switching between SSE and AVX2, unless you do it wrong. What are you referring to specifically?

Are you talking about state transition penalties that can occur if you forget a vzeroupper? That's the only thing I'm aware of which kind of matches that.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#199

Earlier quoted context omitted.

I'm going off topic at this point but I'd think for a native app the main advantages of a binary format would be the static typing and code generation that come from using an IDL.

Rust (the language the Xi core is developed in) has static typing for JSON, as well as other serialization formats: https://github.com/serde-rs/json

I'm familiar with serde. It's an incredible project, but I wouldn't quite call it "static typing for JSON". You still have to unwrap the parse at some point. However, I will concede the point that if you have Rust on both sides then you'll get most of the benefits.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#200

Earlier quoted context omitted.

Not really a question, but if you ever get to the point of wondering what a good next challenging project would be, consider generalizing some of these techniques into a next generation Yacc / Bison replacement. Something that can take generic grammer rules and turn it into a high performance parsing engine. It wouldn't have to support every possible grammar or option. Json isn't that complex of a language, but even…

It's on the list as a research project. It's not obvious to me at this stage that the bottlenecks for more advanced parsers are necessarily going to be in the same place as they are for JSON. It might make more sense to look at a state-of-the-art parser and see if we can contribute a few tricks instead.

I'm just starting to look at Tree-sitter; that might qualify as a state of the art parser that could use a few tricks.
Post reply on HN