Honest question: why not use 2-3 different hash algorithms to minimize collisions if you are simply after uniqueness (e.g.: verifying that a downloaded file is correct)? As for hash tables, can't you guarantee uniqueness by using only reversible operations? I remember reading a lengthy post about this on HN, but can't find the link anymore.
Which hashing algorithm is best for uniqueness and speed?
51–60 of 110 posts
Re: Which hashing algorithm is best for uniqueness and speed?
#52Question: say you use a hash which returns a 32-bit integer. If you were actually implementing a hash table, would you need to declare a structure with 2^32 elements? `int buckets[2^32]`? This seems unwieldy! Would an actual hash table only use, say, the first 10 bits or something of a hash function (int buckets[1024]) to make it less sparse (albeit increase collisions?) If you decide you want more buckets later on,…
Re: Which hashing algorithm is best for uniqueness and speed?
#53FNV despite much popularity is a relatively poor quality hash. Murmur2 has a major flaw, hence Murmur3. CRC32 is slow. Not mentioned in the post, but if you're thinking Fletcher or Adler, they have terrible distribution. For a fast 32-bit hash, much better to go with Bob Jenkin's one-at-a-time hash ( http://en.wikipedia.org/wiki/Jenkins_hash_function#one-at-a-... ) which is simpler than Murmur3 and displays much bett…
Performance graphs: http://www.team5150.com/~andrew/noncryptohashzoo/speed.html
Re: Which hashing algorithm is best for uniqueness and speed?
#54Question: say you use a hash which returns a 32-bit integer. If you were actually implementing a hash table, would you need to declare a structure with 2^32 elements? `int buckets[2^32]`? This seems unwieldy! Would an actual hash table only use, say, the first 10 bits or something of a hash function (int buckets[1024]) to make it less sparse (albeit increase collisions?) If you decide you want more buckets later on,…
(2) Strictly speaking you don't need this overhead and you can use trees to keep it sparse as well, although as far as I know the low-n overhead slows it down too much in practice when compared to the high-n space efficiency. That is, you could declare a binary tree structure with only those buckets and pathways of the 2^32 which you need, but to find something within the structure requires checking and following ~32 pointers and storing something requires creating ~32 nodes.
Re: Which hashing algorithm is best for uniqueness and speed?
#55Earlier quoted context omitted.
The modolo function is pretty much required unless you have 2^32 bytes of RAM available for each hashtable. Modolo is also ridiculously cheap, especially compared to the hashing function.
Certainly, if you are hashing a large amount of data then a single modulo operation isn't going to cost a lot. But if you're just hashing an English word, possibly even a URL, I'm not so sure. You could shift or mask the result (as described above) to use a table size of any power of 2 to avoid the modulo.
Re: Which hashing algorithm is best for uniqueness and speed?
#56Question: say you use a hash which returns a 32-bit integer. If you were actually implementing a hash table, would you need to declare a structure with 2^32 elements? `int buckets[2^32]`? This seems unwieldy! Would an actual hash table only use, say, the first 10 bits or something of a hash function (int buckets[1024]) to make it less sparse (albeit increase collisions?) If you decide you want more buckets later on,…
It depends on your performance use case. 2^32 is only about 540MB of RAM so you can allocate that all at once and then all subsequent actions are read/writes.
Yes, if you're only storing a single bit for each. Usually, you'll want to store a pointer to the head of a linked list. At 64 bits per pointer, you'll need to allocate 32 gigs of RAM - slightly less feasible.
Re: Which hashing algorithm is best for uniqueness and speed?
#57Earlier quoted context omitted.
Computing two 32-bit results in parallel and mixing them at the end does NOT mean collision resistance is only as good as a 32-bit hash. For that, you need to compute ONE 32-bit result, then transform it into a 64-bit result.
Depends whether the two 32-bit hashes are correlated with each other. If there is no correlation then a pair of 32-bit hashes is no more likely to collide than a single 64-bit hash. But this is difficult to achieve, and you should not assume (for example) running the same algorithm twice with different initial states will produce uncorrelated hashes.
Re: Which hashing algorithm is best for uniqueness and speed?
#58http://blog.aggregateknowledge.com/2011/12/05/choosing-a-goo... http://blog.aggregateknowledge.com/2011/12/29/choosing-a-goo... http://blog.aggregateknowledge.com/2012/02/02/choosing-a-goo...
Re: Which hashing algorithm is best for uniqueness and speed?
#59Earlier quoted context omitted.
Given Moore's law and the prevalence of botnets and cloud computing - hash speed really isn't a very strong means of defence against cracking... Also see: http://cyberarms.wordpress.com/2010/10/21/cracking-14-charac...
Uh? Hash speed is pretty much the only means of defense against brute-force cracking (assuming the hash function isn't broken and the attacker has no other vector available). (low) hash speed is the whole point of PBKDF2, bcrypt or scrypt.
See this: http://net.tutsplus.com/tutorials/php/understanding-hash-fun... and: http://stackoverflow.com/questions/1191112/password-hashing-...
to learn more about preventing brute-force attacks
Re: Which hashing algorithm is best for uniqueness and speed?
#60Earlier quoted context omitted.
actually, just as long. Any set of possible bits is a valid input, so the input has one bit of entropy per bit. If the hash were not "just as long" then one of the following must be true: - it collides (two possible inputs would have the same hash) - some inputs could not hash - the algorithm encodes more than 2 bits of entropy per bit. This is not possible by the pigeonhole principle. http://en.wikipedia.org/wiki/Pi…
I was indeed thinking of lossless compression: In practice it compresses almost all its input. If the the output length does not have to be constant, it's fine that some input actually gets inflated. As long as it compresses the average input. But in this case the output length might have to be constant? If that's the case, you are of course correct.