The relevant search term is "stylometry". One particular paper I remember is from Dawn Song's group at Berkeley a couple years back: http://www.cs.berkeley.edu/~dawnsong/papers/2012%20On%20the%... There's a lot of public work on the topic, but it looks like right now the best place to look is still in academic papers (I don't know of any open source libraries, for example).
Ask HN: Algorithms for text fingerprinting?
31–40 of 44 posts
Re: Ask HN: Algorithms for text fingerprinting?
#32Figured I'd chime in here since I developed an algorithm recently that could be applied to this problem with some basic ML. Basically the first step would be shingling the text (choosing a sampling domain) and generating a MinHash struct (computationally cheap) which can then be used to find the "similarity" between sets, or, the "Jaccard Index." If you're clever about this, you can use HyperLogLogs to encode these M…
> "... use HyperLogLogs to encode these MinHash structs gaining a great deal of speed with a marginal error rate, all while allowing for arbitrary N-levels of intersection"
Thanks
Re: Ask HN: Algorithms for text fingerprinting?
#331) tokenize each text into a different bag(set) of words.
2) Compute the Jaccard index[1] using the two sets.
Here's another
1) tokenize each text into a multi-bag(set) of words, keeping track of token frequency
2) keeping the token frequency, order the sets into lists
3) map the lists of words onto an n-dimensional space (where n is say...all of the words into the two documents) as vectors
4) compute the cosine similarity [2]
Here's another:
1) tokenize the texts into two bags of words
2) compute the set difference going both ways.
3) does either difference contain discriminator tokens that rule it out as being from that person
4 (optional)): extend to 2-3-n-grams
Here's another (a variant of the one above):
1) compute 1-2-3-n-grams from one of the texts
2) insert the n-grams into a set
3) compute the same for the second document and test for set membership
4) compute the number of total n-grams from your second document
5) compute (non-in-set/total-n-grams) * 100 to yield a "uniqueness" measure
6) determine if the second document is "unique" enough
And another:
1) assuming you have a sample corpus from a writer and want to know if a new text belongs in that corpus
2) follow the method above but for step #1 and 2 do it with the entire reference corpus
And another:
1) produce an ontology of discriminator terms and categories unique to the writer
2) use an (named entity recognition) NER tool of some kind to find those terms in each document
3) use the set of found terms as an alternative to a bag of words for the Jaccard or Vector models above
You may need to play with stopword list removal, tokenization schemes and n-gram windows (for example, omitting 1-grams might focus the analysis on phrase usage vs. vocabulary usage)
Re: Ask HN: Algorithms for text fingerprinting?
#34Earlier quoted context omitted.
Took only a minute to try: English -> Filipino (Tagalog) -> Chinese-simplified (Mandarin?) -> English I remember reading an article about a year ago (NSA) to identify the user, based on how they are written, vocabulary, spelling errors, grammar, language, and so on. It is interesting to me, because it is difficult to change the written and spoken word in use. It can be estimated that there are between two characters…
Of course you're sending all this information to Google now. Are there any offline translators that are advanced enough to be used for something like this? I imagine most just naively map words 1:1 which wouldn't do much good here.
Re: Ask HN: Algorithms for text fingerprinting?
#35Here's a quick one: 1) tokenize each text into a different bag(set) of words. 2) Compute the Jaccard index[1] using the two sets. Here's another 1) tokenize each text into a multi-bag(set) of words, keeping track of token frequency 2) keeping the token frequency, order the sets into lists 3) map the lists of words onto an n-dimensional space (where n is say...all of the words into the two documents) as vectors 4) com…
Re: Ask HN: Algorithms for text fingerprinting?
#36The relevant search term is "stylometry". One particular paper I remember is from Dawn Song's group at Berkeley a couple years back: http://www.cs.berkeley.edu/~dawnsong/papers/2012%20On%20the%... There's a lot of public work on the topic, but it looks like right now the best place to look is still in academic papers (I don't know of any open source libraries, for example).
"JGAAP is intended to tackle two different problems, firstly to allow people unfamiliar with machine learning and quantitative analysis the ability to use cutting edge techniques on their text based stylometry / textometry problems, and secondly to act as a framework for testing and comparing the effectiveness of different analytic techniques' performance on text analysis quickly and easily."
[1]: http://evllabs.com/jgaap/w/
Looks like there are some other recommendations at http://evllabs.com/jgaap/w/index.php/FAQ#What_other_tools_ar...
Re: Ask HN: Algorithms for text fingerprinting?
#37JStylo that was already mentioned is based on JGAAP. You have some more here: http://evllabs.com/jgaap/w/index.php/FAQ#What_other_tools_ar...
Re: Ask HN: Algorithms for text fingerprinting?
#38Figured I'd chime in here since I developed an algorithm recently that could be applied to this problem with some basic ML. Basically the first step would be shingling the text (choosing a sampling domain) and generating a MinHash struct (computationally cheap) which can then be used to find the "similarity" between sets, or, the "Jaccard Index." If you're clever about this, you can use HyperLogLogs to encode these M…
Could you describe in a little more detail what you mean by this sentence: > "... use HyperLogLogs to encode these MinHash structs gaining a great deal of speed with a marginal error rate, all while allowing for arbitrary N-levels of intersection" Thanks
Let's say we have two sets, and we're trying to find out how similar they are:
setOne = ['the','brown','fox','jumped','a','log']
setTwo = ['the','quick','brown','log','jumped','over','the','fox']
You could use an array intersection when its small, but if you want to do this efficiently at scale you need to take advantage of probabilistic data structures.Let's say you created a HyperLogLog for each set:
setOne = (bitfield representing setOne)
setTwo = (bitfield representing setTwo)
Now HyperLogLogs are cool, because you can merge them together w/o losing anything. You can't retrieve the data, but you can efficiently check if a value exists inside. You can have a false positive, but never a false negative.You might first try simple combinatorics (a la, similarity ~= |A or B| / |A ∪ B|) however this can get hairy depending upon the representation of the HyperLogLog (sparse/dense) and its respective cardinality.
Eventually you realize you can't accept an exponentially-compounding error rate, but still need the raw efficiency, and thus you sacrifice by doubling your storage cost.
Now instead of just the initial sets, you'd also build a parallel MinHash struct:
setOne_minhashes = [, , ]
setTwo_minhashes = [, , ]
setOne = (bitfield representing setOne)
setTwo = (bitfield representing setTwo)
setOneMinHash = (bitfield representation of setOne minhashes)
setTwoMinHash = (bitfield representation of setTwo minhashes)
I won't go into excessive detail about the minhash algorithm itself, but essentially it provides a way to sample a large set of values by selecting/retaining the smallest output hashes. What that means is, you can then intersect the minhash bitfields as many times over as you like, and extract a predictably accurate similarity index in linear time with definable confidence bounds.Re: Ask HN: Algorithms for text fingerprinting?
#39The relevant search term is "stylometry". One particular paper I remember is from Dawn Song's group at Berkeley a couple years back: http://www.cs.berkeley.edu/~dawnsong/papers/2012%20On%20the%... There's a lot of public work on the topic, but it looks like right now the best place to look is still in academic papers (I don't know of any open source libraries, for example).
I'm not sure whether it's appropriate to mention this or not, but I was just looking up the authors of that paper to see what they're doing now. Very sadly, I found out that one of the authors, Emil Stefanov, died last year at the age of 26. http://www.rememberingemil.org
I was a close friend of Emil for a long time. Before he passed, I had gotten busy with work and hadn't spoken to him in a while. I saw some of our mutual friends the weekend before it happened and had planned to call him. Really wish I had made that call sooner.
Re: Ask HN: Algorithms for text fingerprinting?
#40Earlier quoted context omitted.
Could you describe in a little more detail what you mean by this sentence: > "... use HyperLogLogs to encode these MinHash structs gaining a great deal of speed with a marginal error rate, all while allowing for arbitrary N-levels of intersection" Thanks
Sure thing. Let's say we have two sets, and we're trying to find out how similar they are: setOne = ['the','brown','fox','jumped','a','log'] setTwo = ['the','quick','brown','log','jumped','over','the','fox'] You could use an array intersection when its small, but if you want to do this efficiently at scale you need to take advantage of probabilistic data structures. Let's say you created a HyperLogLog for each set: s…
I was initially thinking a directed weighted graph might work well here, but I'm assuming that would scale terribly relative to something like this.