Show HN: High-speed UTF-8 validation in Rust
1–10 of 48 posts
Re: Show HN: High-speed UTF-8 validation in Rust
#2So are there any plans to eventually contribute this to the standard library, e.g. just like hashbrown was?
Re: Show HN: High-speed UTF-8 validation in Rust
#3So are there any plans to eventually contribute this to the standard library, e.g. just like hashbrown was?
[deleted]
Re: Show HN: High-speed UTF-8 validation in Rust
#4So are there any plans to eventually contribute this to the standard library, e.g. just like hashbrown was?
The author discusses this in the corresponding /r/rust thread:
https://www.reddit.com/r/rust/comments/mvc6o5/incredibly_fas...
tl;dr: it's not straightforward, mostly because the standard functionality lives in `core` which doesn't have access to OS support for CPU feature detection.
Re: Show HN: High-speed UTF-8 validation in Rust
#5So are there any plans to eventually contribute this to the standard library, e.g. just like hashbrown was?
[deleted]
Re: Show HN: High-speed UTF-8 validation in Rust
#6One flavour I would expect to be valuable that isn't present here is this:
Process the input (as quickly as possible) and never fail, but replace each invalid sequence of bytes with U+FFFD (bytes 0xEF 0xBF 0xBD).
Re: Show HN: High-speed UTF-8 validation in Rust
#7So are there any plans to eventually contribute this to the standard library, e.g. just like hashbrown was?
The author discusses this in the corresponding /r/rust thread: https://www.reddit.com/r/rust/comments/mvc6o5/incredibly_fas... tl;dr: it's not straightforward, mostly because the standard functionality lives in `core` which doesn't have access to OS support for CPU feature detection.
Does Rust allow for libraries to “override” core functionality when available?
Re: Show HN: High-speed UTF-8 validation in Rust
#8One flavour I would expect to be valuable that isn't present here is this: Process the input (as quickly as possible) and never fail, but replace each invalid sequence of bytes with U+FFFD (bytes 0xEF 0xBF 0xBD).
I bet one could modify this to save indexes of invalid chars and do what you say with just one additional copy
Re: Show HN: High-speed UTF-8 validation in Rust
#9Earlier quoted context omitted.
The author discusses this in the corresponding /r/rust thread: https://www.reddit.com/r/rust/comments/mvc6o5/incredibly_fas... tl;dr: it's not straightforward, mostly because the standard functionality lives in `core` which doesn't have access to OS support for CPU feature detection.
Does Rust allow for libraries to “override” core functionality when available?
If it was all userspace Rust, it would have been possible to choose the implementation at compile time conditional on the presence of std.
Re: Show HN: High-speed UTF-8 validation in Rust
#10One flavour I would expect to be valuable that isn't present here is this: Process the input (as quickly as possible) and never fail, but replace each invalid sequence of bytes with U+FFFD (bytes 0xEF 0xBF 0xBD).
I don't know, you are throwing away the length of the invalid sequence that way. Maybe useful in certain situations, but probably not in most.