Bloom filters don't seem useful as the primary means of deduplication for an actual data storage system to me. - False positives means marking data as duplicate when it's not. - Bloom filters are not associative. So unless the application is very special and can find/ identify /retrieve the data later in some other (likely inefficient?) way, a separate index is required anyway. But if you have a separate index of the…
Yes I originally wrote it for exactly this such a scenario.
This is not for you if you cannot live with any chance of a false positive, even an extremely small one. Note that even in most cases a false positive is acceptable if the chance of a false positive is less then the chance of a hardware error.
Not that just because it is probabilistic does not mean its not useful in many cases. The default setup with 1mil entrie memory and 1/1mil chance false positive chance will run at ~8M mem usage. This is a quite an acceptable trade off in many cases.