Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: High-speed UTF-8 validation in Rust

github.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Show HN: High-speed UTF-8 validation in Rust

#11

Post by nerdponx »

Does Rust compile code that can be used/called from other languages with an FFI? That to me is always one of the persistent advantages of C, I don't have to fully understand how compile machine code works on a technical level, but there are lots of different languages that let me use C libraries in their own language runtimes.

It would be great to have things like high-performance unicode handling with consistent semantics across multiple languages!

Re: Show HN: High-speed UTF-8 validation in Rust

#12

Post by gliptic »

nerdponx wrote: post #11

Does Rust compile code that can be used/called from other languages with an FFI? That to me is always one of the persistent advantages of C, I don't have to fully understand how compile machine code works on a technical level, but there are lots of different languages that let me use C libraries in their own language runtimes. It would be great to have things like high-performance unicode handling with consistent sem…

Yes, you can expose things with the C ABI. This is essential for how it's used in Firefox.

Re: Show HN: High-speed UTF-8 validation in Rust

#13

Post by jhgb »

nerdponx wrote: post #11

Does Rust compile code that can be used/called from other languages with an FFI? That to me is always one of the persistent advantages of C, I don't have to fully understand how compile machine code works on a technical level, but there are lots of different languages that let me use C libraries in their own language runtimes. It would be great to have things like high-performance unicode handling with consistent sem…

A few days ago I read https://dev.to/veer66/calling-rust-from-common-lisp-45c5 , which illustrates presumably what you're talking about.

Re: Show HN: High-speed UTF-8 validation in Rust

#15

Post by hardwaresofton »

nerdponx wrote: post #11

Does Rust compile code that can be used/called from other languages with an FFI? That to me is always one of the persistent advantages of C, I don't have to fully understand how compile machine code works on a technical level, but there are lots of different languages that let me use C libraries in their own language runtimes. It would be great to have things like high-performance unicode handling with consistent sem…

Yes it does -- one of the great parts about Rust. There are lots of projects out there slotting rust into other languages and getting crazy speedups.

- https://doc.rust-lang.org/reference/linkage.html

- https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html#call...

Re: Show HN: High-speed UTF-8 validation in Rust

#16

Post by nezirus »

tialaramex wrote: post #6

One flavour I would expect to be valuable that isn't present here is this: Process the input (as quickly as possible) and never fail, but replace each invalid sequence of bytes with U+FFFD (bytes 0xEF 0xBF 0xBD).

Yeah, I'd like to see that as well, something like to_string_lossy(), but really fast, so I can call it whenever I expect occasional broken UTF-8 string (and don't care about few replaced characters).

Re: Show HN: High-speed UTF-8 validation in Rust

#19

Post by tyingq »

meisel wrote: post #17

How does this compare, speed-wise, to the UTF-8 validation done in simdjson?

If someone is wanting to make a code comparison, most of the UTF-8 validation in simdjson seems to be nicely summed up in this pull request that sped it up: https://github.com/simdjson/simdjson/pull/993/files

Re: Show HN: High-speed UTF-8 validation in Rust

#20

Post by chrismorgan »

tialaramex wrote: post #6

One flavour I would expect to be valuable that isn't present here is this: Process the input (as quickly as possible) and never fail, but replace each invalid sequence of bytes with U+FFFD (bytes 0xEF 0xBF 0xBD).

One major problem with this: you can’t do it in place. Invalid byte sequences could be 1–4 bytes long, but U+FFFD is exactly three bytes long.
Post reply on HN