Previous blog post on HN:
Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
21–30 of 54 posts
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#22Under the new string model in java > 8 a fairly frequent workflow is: 1) get external string 2) figure out if it is UTF-8, UTF-16, or some other recognizable encoding 3) validate the byte stream 4) figure out if the code points in the incoming string can be represented in Latin-1 5) instantiate a java string using either the Latin-1 encoder or the UTF-16 encoder I know some or all of these steps are done using hotspo…
You might be interested in his blog on the same subject a few days ago:
https://lemire.me/blog/2018/10/16/validating-utf-8-bytes-jav...
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#23Earlier quoted context omitted.
Nice. So I understand that AVX2 is not bringing the CPU's clock down. Got any sources for power consumption figures/comparisons of those AVX units?
Heavy use of complex AVX2 operations causes downclocking, too, but typically less so than AVX-512. More details are documented in https://en.wikichip.org/wiki/intel/frequency_behavior -- also see e.g. https://en.wikichip.org/wiki/intel/xeon_gold/6138#Frequencie... for an example how the frequencies differ depending on the number of active cores. I think the reason for reducing clock speed when vector units are in hea…
It's worth noting that the cloudflare test was done on a Xeon Silver, which has worse properties around the frequency changes than the Gold or Platinum. If you're on either Gold or Platinum, you're less likely to suffer the problems that Cloudflare did with mixed workloads.
This seems an optimisation nightmare. Your program needs to be aware both of the capability of the chip for using instructions, and what type of chip it is within a family to decide if you maybe do or don't want to use certain vectored instructions.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#24I see a lot of applications trying to take advantage of SIMD, but what when you try to run them on systems that don't support these instructions? My guess is that you need to write multiple files taking advantage of different sets of instructions and then dynamically figure out which to use at runtime with cpuid, but isn't that cumbersome and a way to inflate a codebase dramatically?
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#25I see a lot of applications trying to take advantage of SIMD, but what when you try to run them on systems that don't support these instructions? My guess is that you need to write multiple files taking advantage of different sets of instructions and then dynamically figure out which to use at runtime with cpuid, but isn't that cumbersome and a way to inflate a codebase dramatically?
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#26Earlier quoted context omitted.
What was the size of the SQL file? A "few table names" doesn't mean much if the SQL file is 20GB. In any case, sed and awk are plenty fast, but not the fastest methods of text manipulation. You could write a custom C program for that.
While it sure is possible to do text manipulation in C, I don't think it should ever be the first choice, even if 'fastest' is a goal. A 0 byte is perfectly acceptable in a utf8 string (or any unicode string, really). But C has those annoying zero-terminated strings, so if you want to manipulate arbitrary unicode strings the first thing you can do is kiss the string functions in the C standard library goodbye. Which…
> A 0 byte is perfectly acceptable in a utf8 string (or any unicode string, really)
What? My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL. That all normal human language described by a utf8 string will never contain a NUL. They're comparable to C strings in that way, where it can be used safely as an end of string marker. If you have embedded NULs, it's not really utf8, is it?
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#27Earlier quoted context omitted.
While it sure is possible to do text manipulation in C, I don't think it should ever be the first choice, even if 'fastest' is a goal. A 0 byte is perfectly acceptable in a utf8 string (or any unicode string, really). But C has those annoying zero-terminated strings, so if you want to manipulate arbitrary unicode strings the first thing you can do is kiss the string functions in the C standard library goodbye. Which…
> Which you probably want to do anyway because pascal-strings are simply better. They're not though. While having an explicit length is great, p-strings means the length is the first item of the data buffer, which is just awful, and why Pascal was originally limited to 255 byte strings. Rust or C++ use record-strings, where the string type is a "rich" stack-allocated structure of (*buffer, length[, capacity], …) rath…
> p-strings means the length is the first item of the data buffer, which is just awful
You can represent it as a struct of (length, char[]) which isn't awful.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#28Earlier quoted context omitted.
> Which you probably want to do anyway because pascal-strings are simply better. They're not though. While having an explicit length is great, p-strings means the length is the first item of the data buffer, which is just awful, and why Pascal was originally limited to 255 byte strings. Rust or C++ use record-strings, where the string type is a "rich" stack-allocated structure of (*buffer, length[, capacity], …) rath…
> p-strings means the length is the first item of the data buffer, which is just awful You can represent it as a struct of (length, char[]) which isn't awful.
[deleted]
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#29Under the new string model in java > 8 a fairly frequent workflow is: 1) get external string 2) figure out if it is UTF-8, UTF-16, or some other recognizable encoding 3) validate the byte stream 4) figure out if the code points in the incoming string can be represented in Latin-1 5) instantiate a java string using either the Latin-1 encoder or the UTF-16 encoder I know some or all of these steps are done using hotspo…
If you are given the external string as bytes, which is all you can have if you don't know the encoding. Then steps 2,3,4 can all be done as one step I would have thought. Something like - https://github.com/adamretter/utf8-validator/blob/optimize-u...
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#30Earlier quoted context omitted.
While it sure is possible to do text manipulation in C, I don't think it should ever be the first choice, even if 'fastest' is a goal. A 0 byte is perfectly acceptable in a utf8 string (or any unicode string, really). But C has those annoying zero-terminated strings, so if you want to manipulate arbitrary unicode strings the first thing you can do is kiss the string functions in the C standard library goodbye. Which…
> A 0 byte is perfectly acceptable in a utf8 string (or any unicode string, really) What? My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL. That all normal human language described by a utf8 string will never contain a NUL. They're comparable to C strings in that way, where it can be used safely as an end of string marker. If you have embedded NULs, it's not r…
> My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL.
Correct.
> That all normal human language described by a utf8 string will never contain a NUL.
Correct.
> If you have embedded NULs, it's not really utf8, is it?
Incorrect.
NUL is a valid character. If you accept arbitrary utf-8, or arbitrary ascii, or arbitrary 8859-1, then there might be embedded NUL. You can filter them out if you want, but they're not invalid.