21–30 of 53 posts
Modern Intel CPUs have an instruction specifically to compute CRC. This instruction is easily consumed through a C++ intrinsic, literally in one line of code. You can't do any better than that, no matter what you use.
Modern Intel CPUs have an instruction specifically to compute CRC. This instruction is easily consumed through a C++ intrinsic, literally in one line of code. You can't do any better than that, no matter what you use.
[Intel's CRC32 CPU instructions] uses different inputs to the CRC32 algorithm (known as the polynomial) which is apparently more robust, and is used in networks, filesystems, that sort of thing. It gives different results to the "standard" polynomial, typically used in compression.
They would have to go back and recompute all their existing stored checksums.
Also, literature suggests a throughput of ∼2.67 bytes per cycle for the CRC32 instruction, a three fold improvement over best in class non-HW routines. I'm quite sure it would be worth it to reconvert previous checksums if you can do so in a way that minimizes downtime (think TrueCrypt doing a transparent initial encryption; not encrypting when theres IO load).
why not rip out CRC32 and put in xxhash?
Modern Intel CPUs have an instruction specifically to compute CRC. This instruction is easily consumed through a C++ intrinsic, literally in one line of code. You can't do any better than that, no matter what you use.
They have lots of data hashed against a different CRC polynomial. The CRC32C acceleration instructions are unsuitable.
why not rip out CRC32 and put in xxhash?
Last time I needed really fast hashing I used FNV. How does it compare to xxhash?
Earlier quoted context omitted.
Last time I needed really fast hashing I used FNV. How does it compare to xxhash?
Never mind - it's covered in the other reply link here. Looks like xxhash is the goto hash algorithm now.
Earlier quoted context omitted.
Never mind - it's covered in the other reply link here. Looks like xxhash is the goto hash algorithm now.
That's a misunderstanding. If you need a hash or checksum for very long data (megabytes, gigabytes), use xxhash. It's not the best for hash tables.
If they have CRC accounting for 10% of CPU, they must be using these checksums a lot. At some point I'd imagine the false error rate simply due to bit flips and other random errors on the path from database through CRC function will outlast whatever value you are getting from the constant rechecks. Also, literature suggests a throughput of ∼2.67 bytes per cycle for the CRC32 instruction, a three fold improvement over…