31–35 of 35 posts
I've used this technique before for image searching, here's some tips: The algorithm presented here can be summarized as "Take the low-frequency DCT components of the luminance channel, quantized down to one bit". Hashing is a very misleading term for what you get from it: In particular, it doesn't have anything like the uniform coverage of the output space you would expect from a hash function, as almost all encount…
This made me think of something, so bear with me for a second. I took Andrew Ng's Machine Learning class on Coursera, and one of the things he talked about was how, given a certain data set and a logistic regression approach, the algorithm might decide to just always give it a value of 1, since 98% of the time the value is one. The examples given here were all predicted to have 95%+ similarity, but it seems to me tha…
In this case, we are not looking at a model training, we are just evaluating the outcome of a (static) algorithm. On the other hand, in the context of ML you should try to include as much information as possible in your training dataset. This is a concept that comes by in many fields, some of them just tangentially related to machine learning (e.g. excitation signals for system identification, speaking both in terms of frequency domain and differential equations).
tl;dr: there is no "training set" to speak in the presented article - hence, concepts such as overfitting and information content (of the training set) do not quite apply here.
This made me think of something, so bear with me for a second. I took Andrew Ng's Machine Learning class on Coursera, and one of the things he talked about was how, given a certain data set and a logistic regression approach, the algorithm might decide to just always give it a value of 1, since 98% of the time the value is one. The examples given here were all predicted to have 95%+ similarity, but it seems to me tha…
While your comment applies on a broader sense (we need to test both for cases in which we expect the algorithm to find similarity and cases in which we expect to find dissimilarity - lest we embrace confirmation bias), I believe your reference to machine learning is a bit off. In this case, we are not looking at a model training, we are just evaluating the outcome of a (static) algorithm. On the other hand, in the co…
Earlier quoted context omitted.
Hm. Well, I want to hash the pattern of bits in the text file, like a cryptographic hash does (suppose md5 or sha-1 for simplicity's sake). I'll provide some examples of input and output. These examples happen to contain no linefeeds. Suppose: Hello, World! --> 65a8e27d8879283831b664bd8b7f0ad4 Then I want something like: Hello, Worlds! --> 65a8e27d8879283831b664bd8b7f4ad4 ...Rather than what md5 currently provides: H…
Perhaps I'm missing the point, but I came up with a naive and sloppy part solution: https://gist.github.com/peterc/737d9178f02118f8e315 .. there are some weaknesses to this solution but it does perform similarly to your examples.
Do you (or anyone else) have time to describe this code in English or pseudocode?
I'll start.
Let there be a function called hash which takes a
string named str and an integer named
length_of_hash (which defaults to 20).
Slice the string (str) into length_of_hash equally-sized
pieces?
Take numeric value of each character (of each slice??)
and do.. something to it, something involving
modulo 256, unless it's zero. Save all the
results.
Express each numeric result as hexidecimal and append
all those together. Return it, probably.
Earlier quoted context omitted.
Perhaps I'm missing the point, but I came up with a naive and sloppy part solution: https://gist.github.com/peterc/737d9178f02118f8e315 .. there are some weaknesses to this solution but it does perform similarly to your examples.
Thanks, Peter. I don't speak Ruby. Uh, line 5 contains a small function definition, which is applied to each ...slice of the input string, is that right? Do you (or anyone else) have time to describe this code in English or pseudocode? I'll start. Let there be a function called hash which takes a string named str and an integer named length_of_hash (which defaults to 20). Slice the string (str) into length_of_hash eq…
Example, with a hash length of 4: "helloworld" => [104, 101, 108, 108, 111, 119, 111, 114, 108, 100] => [[104, 101, 108, 108], [111,119,111,114], [108, 100]] => [67, 64, 219, 222] => "4340dbde"
So if one character is different in the string, only one hex value in the output will vary too. However, a flaw of the plan is that changes spaced out at an interval that matches your hash "length" will only affect one hex value in the output, but you run into the pigeonhole principle if you want a hash of limited size to have the same or similar Levenshtein distance as the potential inputs, but I suspect there are far smarter solutions :-)