Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
1–10 of 54 posts
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#2What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#3What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.
I don't think they use anything in common. Try to set your locale to "C" as otherwise string comparisons will do extra work handling your locale's notions of equivalent characters.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#4What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.
Note that this and that are not necessarily related: you're talking about performing unicode-aware text matching and manipulation, TFA is solely about validating a buffer's content as UTF-8.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#5What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.
How slow? On my 2013 MBP, `gsed` (sed from coreutils) can do a replacement like that at about 350 MiB/s (of which most seems to be spent writing to disk, since writing to /dev/null hikes it up to 800 MiB/s).
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#6What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.
How slow? On my 2013 MBP, `gsed` (sed from coreutils) can do a replacement like that at about 350 MiB/s (of which most seems to be spent writing to disk, since writing to /dev/null hikes it up to 800 MiB/s).
It was sed substitute command on a ~800Mb file on Thinkpad T470 with SSD. It was taking around 40-50 sec for each substitution. Though as others have pointed, it may not be directly related to article in discussion.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#7What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.
What was the size of the SQL file?
A "few table names" doesn't mean much if the SQL file is 20GB.
In any case, sed and awk are plenty fast, but not the fastest methods of text manipulation. You could write a custom C program for that.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#8Earlier quoted context omitted.
How slow? On my 2013 MBP, `gsed` (sed from coreutils) can do a replacement like that at about 350 MiB/s (of which most seems to be spent writing to disk, since writing to /dev/null hikes it up to 800 MiB/s).
It was sed substitute command on a ~800Mb file on Thinkpad T470 with SSD. It was taking around 40-50 sec for each substitution. Though as others have pointed, it may not be directly related to article in discussion.
>It was taking around 40-50 sec for each substitution.
Substitution should not be really a relevant metric as it wouldn't influence the result much. Sed/Awk will still have to go through the whole file to find all occurrences they should substitute (and when they do find an occurrence, the substitution would take nanoseconds).
The size of the file is a better metric (e.g. how many seconds for that 800mb in total).
Also, whether you used regex in your awk/sed, and what kind. A badly written regex can slow down search very much.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#9I wonder about the Joules per byte. AFAIK AVX units are quite expensive energy-wise.
Re: Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)
#10I wonder about the Joules per byte. AFAIK AVX units are quite expensive energy-wise.
Don't they also tend to work at a lower clock due to their higher energy requirements?
edit: though this is AVX2 ("AVX-256") rather than AVX-512, and Lemire has covered AVX and the possibility of throttling (with or without AVX) in the past so they're probably aware of the potential issue and consider that they either won't get triggered or the gain is good enough to compensate the lower frequency.