Not sure how the author of this entry on HN managed to change original title from gigabytes per second to gigabits per siemens :)
Probably auto-capitalization gone wrong. Or some very new code ;)
Leveraging SIMD: Splitting CSV Files at 3Gb/S
11–20 of 43 posts
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#12Not sure how the author of this entry on HN managed to change original title from gigabytes per second to gigabits per siemens :)
Autocorrector issues + fast fingers to click on submit without double checking. Sorry for that.
Whoever fixed the title, thank you :D
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#13Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#14Not sure how the author of this entry on HN managed to change original title from gigabytes per second to gigabits per siemens :)
Autocorrector issues + fast fingers to click on submit without double checking. Sorry for that. Whoever fixed the title, thank you :D
> fixed the title
It still shows as "3Gb/S" for me, instead of "3GB/s"
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#15Nice, but I'm afraid real world CSVs are a lot more complicated than described so don't use this code in production.
If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format ( ) it can definitely make sense to use a high-speed optimized ingester. One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which me…
Perl's best known library Terxt::CSV has both a pure-perl and a C implementation.
Here is the C version
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#16Not sure how the author of this entry on HN managed to change original title from gigabytes per second to gigabits per siemens :)
Isn't it easier to write 3GbΩ instead of 3Gb/S?
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#17Pretty similar article from very recently: https://nullprogram.com/blog/2021/12/04/ Discussion: https://news.ycombinator.com/item?id=29439403 The article mentions in an addendum (and BeeOnRope also pointed it out in the HN thread) a nice CLMUL trick for dealing with quotes originally discovered by Geoff Langdale. That should work here for a nice speedup. But without the CLMUL trick, I'd guess that the unaligned loads…
Hi, I'm one of the authors of the post
Thanks for pointing us to CLMUL, I'm not familiar with these kind of multiplications, but, converting the quote bitmask to a quoted bitmask would certainly make it faster. With this new bitmask, we could negate it and AND it with the newline mask, generating a mask of newlines that are not inside quotes. Getting the last newline then would be a simple CLZ of that mask. And there wouldn't be a need to resort to byte to byte processing.
In our tests, going byte to byte for more iterations to keep the alignment when hitting the "else case" performed worse than making the unaligned loads, but as you say "just use CLMUL" (as all loads will be aligned) :D
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#18Pretty similar article from very recently: https://nullprogram.com/blog/2021/12/04/ Discussion: https://news.ycombinator.com/item?id=29439403 The article mentions in an addendum (and BeeOnRope also pointed it out in the HN thread) a nice CLMUL trick for dealing with quotes originally discovered by Geoff Langdale. That should work here for a nice speedup. But without the CLMUL trick, I'd guess that the unaligned loads…
Hi, I'm one of the authors of the post Thanks for pointing us to CLMUL, I'm not familiar with these kind of multiplications, but, converting the quote bitmask to a quoted bitmask would certainly make it faster. With this new bitmask, we could negate it and AND it with the newline mask, generating a mask of newlines that are not inside quotes. Getting the last newline then would be a simple CLZ of that mask. And there…
PMOVMSKB/BSF/POPCNT takes serious wizardry, but instructions like PCLMULLQLQDQ make you feel like Gandalf. It's defined:
pair clmul(uint64_t a, uint64_t b) {
uint64_t t, x = 0, y = 0;
if (a && b) {
if (bsr(a) >= 1) {
if (b & 1) x ^= a, y ^= t;
t = t > 63;
}
}
return (pair){x, y};
}
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#19Pretty similar article from very recently: https://nullprogram.com/blog/2021/12/04/ Discussion: https://news.ycombinator.com/item?id=29439403 The article mentions in an addendum (and BeeOnRope also pointed it out in the HN thread) a nice CLMUL trick for dealing with quotes originally discovered by Geoff Langdale. That should work here for a nice speedup. But without the CLMUL trick, I'd guess that the unaligned loads…
Hi, I'm one of the authors of the post Thanks for pointing us to CLMUL, I'm not familiar with these kind of multiplications, but, converting the quote bitmask to a quoted bitmask would certainly make it faster. With this new bitmask, we could negate it and AND it with the newline mask, generating a mask of newlines that are not inside quotes. Getting the last newline then would be a simple CLZ of that mask. And there…
CLMUL in general is a bit weird to wrap your head around, but a CLMUL with -1 isn't too tricky: it's like a running 1-bit sum, or in other words, each bit in the result is the parity of all the bits up to that point in the multiplier.
> In our tests, going byte to byte for more iterations to keep the alignment when hitting the "else case" performed worse than making the unaligned loads, but as you say "just use CLMUL" (as all loads will be aligned) :D
I was talking about using bitwise operations with the quote/escape/newline masks already computed (like in the blog post I linked), rather than a byte-by-byte loop. But yeah, CLMUL is better anyways :)
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#20Not sure how the author of this entry on HN managed to change original title from gigabytes per second to gigabits per siemens :)
Staying with Physics, "Gb/S" is Gigabarns per Siemens. Some relation of electrical conductance with cross-sectional area.
The barn is a unit of cross-sectional area, based on the Uranium nucleus (area 1 barn). Uranium is pretty large in atomic terms; the name is from the idiom "couldn't hit the broad side of a barn".