Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: High-speed UTF-8 validation in Rust

github.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Show HN: High-speed UTF-8 validation in Rust

#41

Post by tialaramex »

cryptonector wrote: post #38
chrismorgan wrote: post #20

Earlier quoted context omitted.

One major problem with this: you can’t do it in place. Invalid byte sequences could be 1–4 bytes long, but U+FFFD is exactly three bytes long.

Use an ASCII character like `?` or, even better, ASCII SUB (0x1A, substitute), which is specifically intended for this sort of thing. Failing that, there are a number of other unused ASCII control characters like VT (vertical tab), FS (file separator), GS (group separator), US (unit separator), CAN (cancel), etc.

No. Don't do any of these things. The reason U+FFFD exists, even though ASCII has any number of fun things you can scribble in one byte is that U+FFFD specifically isn't any of the things your program probably didn't expect to appear unexpectedly after unrelated processing.

It isn't a letter, or a digit, or whitespace, or punctuation, or a word separator, or a control character, it is neither uppercase nor lowercase, it doesn't have any canonical equivalents - it's just a codepoint that exists specifically for this purpose.

As a result it's much less likely that if gibberish sneaks into your system somehow and gets turned into U+FFFD this causes something important to break elsewhere.

And when sooner or later a human is shown this text, it's very obvious that U+FFFD isn't what they expected, whether that was E-acute, a Euro currency symbol, a cat emoji or whatever else, and the human will know something went wrong and can decide if they care about that.

Re: Show HN: High-speed UTF-8 validation in Rust

#43

Post by burntsushi »

cryptonector wrote: post #39
jfk13 wrote: post #14

I'd be curious how this compares with the encoding_rs crate ( https://github.com/hsivonen/encoding_rs ), which also offers UTF-8 validation (among many other things).

This is SIMD based. If the encoding_rs crate isn't SIMD based, then this thing will beat its pants off.

It does use SIMD (when enabled). But also, the encoding_rs crate is doing more. It isn't just checking for validity. It's also doing transcoding and error detection.

Re: Show HN: High-speed UTF-8 validation in Rust

#44

Post by cryptonector »

tialaramex wrote: post #41
cryptonector wrote: post #38

Earlier quoted context omitted.

Use an ASCII character like `?` or, even better, ASCII SUB (0x1A, substitute), which is specifically intended for this sort of thing. Failing that, there are a number of other unused ASCII control characters like VT (vertical tab), FS (file separator), GS (group separator), US (unit separator), CAN (cancel), etc.

No. Don't do any of these things. The reason U+FFFD exists, even though ASCII has any number of fun things you can scribble in one byte is that U+FFFD specifically isn't any of the things your program probably didn't expect to appear unexpectedly after unrelated processing. It isn't a letter, or a digit, or whitespace, or punctuation, or a word separator, or a control character, it is neither uppercase nor lowercase,…

This is about what can be done in SIMD. See GP and GGP. I think using ASCII SUB is a perfectly reasonable thing to do in this context. You can always further post-process the result to turn ASCII SUB into U+FFFD.

Re: Show HN: High-speed UTF-8 validation in Rust

#45

Post by pgwhalen »

monocasa wrote: post #30
celeritascelery wrote: post #27

I find it interesting that with the state of rust SIMD many implementations just opted for reimplementing the code multiple times for each intrinsic (SSE, Neon, AVX, etc). One would think that one of the generic libraries (faster, simdeez, packed_simd) would start to see widespread use, but they all seem to have issues that have prevented adoption.

The various SIMD ISAs are different enough from eachother that you really want a custom implementation per ISA for perf reasons. Particularly if you're doing something off the beaten path like this rather than cranking through some vector math.

Can you comment on the Vector API that Java is set to include? Do you think it's entirely misguided, or more of a best effort given what one can reasonably expect from Java?

http://openjdk.java.net/jeps/8261663

(I'm completely ignorant on the subject)

Re: Show HN: High-speed UTF-8 validation in Rust

#47

Post by kryps »

burntsushi wrote: post #43
cryptonector wrote: post #39

Earlier quoted context omitted.

This is SIMD based. If the encoding_rs crate isn't SIMD based, then this thing will beat its pants off.

It does use SIMD (when enabled). But also, the encoding_rs crate is doing more. It isn't just checking for validity. It's also doing transcoding and error detection.

As far as I can see it does not use SIMD for UTF8 validation, only likely/unlikely instrinsics, see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...

Re: Show HN: High-speed UTF-8 validation in Rust

#48

Post by jfk13 »

kryps wrote: post #47
burntsushi wrote: post #43

Earlier quoted context omitted.

It does use SIMD (when enabled). But also, the encoding_rs crate is doing more. It isn't just checking for validity. It's also doing transcoding and error detection.

As far as I can see it does not use SIMD for UTF8 validation, only likely/unlikely instrinsics, see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...

That's not the only use of SIMD in the crate (e.g. see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...), but I haven't looked into exactly where/how it's used further.
Post reply on HN