Live data from Hacker News

Fast Perfect Hashing

jandrewrogers.com

11–20 of 43 posts

Re: Fast Perfect Hashing

#11

this..seems extremely cool (?) does someone have a reference or pointer for further study about how this is 'perfect' and not just 'really good'

A perfect hash function has no collisions. Typically, people are interested in perfect hashes constructed on types that themselves are not represented in the number of output bits but with an input cardinality that is less than or equal to the output's cardinality. What the author is describing here is more commonly called a mixing function. There are many fast mixing functions. https://gist.github.com/badboy/6267743

Great link, thanks

Re: Fast Perfect Hashing

#12
post #8

> A perfect hash function is one that is collision-free. By implication, the hash must be at least as many bytes as the key I don't understand this lemma. An indexed array is a degenerate example in which the "hash" (pointer) is presumably a lot smaller than the key (index). BTW for those not familiar with a perfect hash function they are great for things like symbol tables that are immutable at runtime. You can make…

It's smaller than the key, but the size of the domain is the number of items you've put in the array.

Re: Fast Perfect Hashing

#13
post #12
post #8

> A perfect hash function is one that is collision-free. By implication, the hash must be at least as many bytes as the key I don't understand this lemma. An indexed array is a degenerate example in which the "hash" (pointer) is presumably a lot smaller than the key (index). BTW for those not familiar with a perfect hash function they are great for things like symbol tables that are immutable at runtime. You can make…

It's smaller than the key, but the size of the domain is the number of items you've put in the array.

Actually, I might have misunderstood you. In your example, the index in the array is treated like the hash of the item you've put in there, why are we suddenly considering pointers?

Re: Fast Perfect Hashing

#14

The OP seems to be confusing terminology. What they construct is a permutation. Everything they say is relevant to permutations, not perfect hashing. In particular: > A perfect hash function is one that is collision-free. By implication, the hash must be at least as many bytes as the key This is just false. Perfect hashing is often if not always performed on subsets of the full domain, e.g. the keywords in a programm…

It should be obvious here that the domain is e.g. integral types on computers. The term is used in many places to denote collision-free quasi-randomization, which technically is a strict subset of perfect hash functions, and I chose it to align with that common usage.

Re: Fast Perfect Hashing

#15
As already mentionend, what he describes should be called a bijective mixing function. Now when i am given n different values, i can mix them them with his function and will get n other pairwise different values. That helps me nothing in practive with regard to the task of perfect hashing. Technically i would say, he describes in deed a perfect hashing function, but of questionable quality.

Re: Fast Perfect Hashing

#16

The OP seems to be confusing terminology. What they construct is a permutation. Everything they say is relevant to permutations, not perfect hashing. In particular: > A perfect hash function is one that is collision-free. By implication, the hash must be at least as many bytes as the key This is just false. Perfect hashing is often if not always performed on subsets of the full domain, e.g. the keywords in a programm…

Yeah the terminology is a bit misleading, it would be better to call this a "bijective" or "invertible" integer hash function, or mixer.

It's really nice work though! Invertible mixers are usually building blocks for more general hash function, for example MurmurHash3 uses this 64-bit mixer

https://github.com/aappleby/smhasher/blob/master/src/MurmurH...

Using this trick could produce a faster murmurhash-type function.

Re: Fast Perfect Hashing

#17

this..seems extremely cool (?) does someone have a reference or pointer for further study about how this is 'perfect' and not just 'really good'

The first sentence of the article explains what a perfect hash function is : "A perfect hash function is one that is collision-free.". Essentially, a perfect hash function is one where no two domain elements map to the same range element. If you want to explore it further, you can search for injective functions and one-to-one mapping. An example of a function that produces collisions is the ABS function. ABS( 2 ) = 2…

The identity function is collision free. It's the fastest one.

Re: Fast Perfect Hashing

#18
This article is interesting but... weird. It seems to assume a bunch of things, and then states some conclusions based on that, but then also assumes the value of the conclusions is obvious.

> A perfect hash function is one that is collision-free. By implication, the hash must be at least as many bytes as the key and the function is theoretically reversible, though not always tractably so.

This is true because of the pigeonhole principle, but most people won't assume that the key size is equivalent to the domain size. Usually when using perfect hashing, you are hashing (much) fewer elements than the total range a key can represent.

A perfect hash function that uniquely assigns hash values to the eight items you need to store, but gives you back integers anywhere in the 32 bit range isn't super helpful. Or, at least, it's not obvious to me why it would be. They do state the algorithm can be scaled down to 8 or 16 bits, so I guess there's that.

> Because the hash is no smaller than the key, the primary use case is randomizing small values like integral types.

This just doesn't make sense to me. The key can be much larger than the hash, even with perfect hashing, as long as the number of keys in your data set is smaller than the maximum hash value.

The author says "primary use case" but what they really mean is that this isn't what most people would consider a perfect hash at all. It's just an integer permuter. In other words, here as an even faster implementation of what the author would describe as a perfect hash over small integer keys:

    uint32_t hash(uint32_t key) { return key; }
This also guarantees there are no collisions so is a "perfect hash". It's just not a very useful one.

The author assumes the reader knows the value of randomizing your integer keys. I'm not sure what that value is in cases where you know you don't have collisions anyway.

Re: Fast Perfect Hashing

#19
post #8

> A perfect hash function is one that is collision-free. By implication, the hash must be at least as many bytes as the key I don't understand this lemma. An indexed array is a degenerate example in which the "hash" (pointer) is presumably a lot smaller than the key (index). BTW for those not familiar with a perfect hash function they are great for things like symbol tables that are immutable at runtime. You can make…

A really common pattern in many programs is to associate metadata with something using a unique key.

In cases where you can’t associate that metadata directly using a pointer (by using structures as keys embedding the pointer in the key) or a direct index into a metadata array, a perfect hash generated at build time is about as good as you’re going to get.

The other thing is that often this is easy to bolt on after the fact. As an example, in network devices it is common for a syslog mnemonic to be associated with an instance of a rate limiter. A good solution is to compile your log messages using an errmsg definition and compiler tool (which also allows you to build documentation) but this requires a lot of up-front planning and build orchestration and makes the code awkward in sone circumstances. An alternative is that you allow people to declare their mnemonic (and, for example, rate limit) in-place and then allow the logger library to use that to look up other properties that are joined from other sources (if there are any) and glean the structures that need to be created.

Since extracting the full set on mnemonics is necessary anyway (for tripwires, for documentation, ...) you can now build the hash function after-the-fact. This definitely simplified the build considerations (and avoids unnecessary rebuilds when, for example, only documentation changes).

The after-the-factness-but-with-low-overhead part is what’s nice about it. Yes, it’s better to structure things up front to make the system optimal but it’s nice to be able to get close.

Re: Fast Perfect Hashing

#20
post #13
post #12

Earlier quoted context omitted.

It's smaller than the key, but the size of the domain is the number of items you've put in the array.

Actually, I might have misunderstood you. In your example, the index in the array is treated like the hash of the item you've put in there, why are we suddenly considering pointers?

In this degenerate case the "hash function" is base_address+(index*sizeof(memory_address))
Post reply on HN