Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition) • Hacker News Board

Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)

#51

Post by wmu » Sun, Oct 21, 2018, 4:16 PM UTC

wmu wrote: post #41

the_clarence wrote: post #24

I see a lot of applications trying to take advantage of SIMD, but what when you try to run them on systems that don't support these instructions? My guess is that you need to write multiple files taking advantage of different sets of instructions and then dynamically figure out which to use at runtime with cpuid, but isn't that cumbersome and a way to inflate a codebase dramatically?

Speaking of the Intel world it's not that bad. There are three major version right now: SSE4.1, AVX and AVX2 (AVX512 is not popular yet). In the past (roughly 10 years ego) it was a problem, as there were: MMX, SSE, SSE2, SSE3, SSSE3 , SSE4.1, SSE4.2, XOP, 3DNow and perhaps a few more extensions. it's not a typo, there are three 'S' :)

Sorry, I forgot that in HN comments the asterisk char is an italics indicator. There should be a mark after SSSE3.

Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)

#52

Post by paulddraper » Sun, Oct 21, 2018, 6:39 PM UTC

paavoova wrote: post #42

Dylan16807 wrote: post #30

Earlier quoted context omitted.

> My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL. Correct. > That all normal human language described by a utf8 string will never contain a NUL. Correct. > If you have embedded NULs, it's not really utf8, is it? Incorrect. NUL is a valid character. If you accept arbitrary utf-8, or arbitrary ascii, or arbitrary 8859-1, then there might be embedded NUL. You c…

It's invalid for unix filenames to have a null character. Therefore, if your application is printing filenames in their unicode representation, it doesn't ever need to consider there to be a null byte. This of course isn't an arbitrary case, but it shows one can make assumptions regardless of the "validity" of a character. I believe for most cases of arbitrary input, the correct and safe thing to do is to assume a by…

It's also invalid for filenames to have a slash, but I don't think that's very relevant to the discussion at hand.

Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)

#53

Post by BeeOnRope » Sun, Oct 21, 2018, 11:28 PM UTC

eesmith wrote: post #38

why_only_15 wrote: post #36

Earlier quoted context omitted.

In my understanding when you use intrinsics and build for a processor without support for the intrinsics then GCC for example will replace it with equivalent code.

That is true. Here's a couple of negatives. First, you still need to build once for each architecture, either as different executables, or as different object files, and provide some dispatch mechanism to use the right one based on what hardware is available. Second, if the intrinsics aren't built-in then there may be faster alternatives than using the GCC emulated version.

You must be thinking about GCC "builtins" because there is no emulation for x86 SIMD intrinsics (ie the things in ).

Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)

#54

Post by eesmith » Tue, Oct 23, 2018, 1:06 PM UTC

BeeOnRope wrote: post #53

eesmith wrote: post #38

Earlier quoted context omitted.

That is true. Here's a couple of negatives. First, you still need to build once for each architecture, either as different executables, or as different object files, and provide some dispatch mechanism to use the right one based on what hardware is available. Second, if the intrinsics aren't built-in then there may be faster alternatives than using the GCC emulated version.

You must be thinking about GCC "builtins" because there is no emulation for x86 SIMD intrinsics (ie the things in ).

Oh, indeed I was. Thanks for pointing out my error. I was specifically thinking about POPCNT.