Live data from Hacker News

Ask HN: Algorithms for text fingerprinting?

news.ycombinator.com

31–40 of 44 posts

Re: Ask HN: Algorithms for text fingerprinting?

#31
post #9

The relevant search term is "stylometry". One particular paper I remember is from Dawn Song's group at Berkeley a couple years back: http://www.cs.berkeley.edu/~dawnsong/papers/2012%20On%20the%... There's a lot of public work on the topic, but it looks like right now the best place to look is still in academic papers (I don't know of any open source libraries, for example).

I'm not sure whether it's appropriate to mention this or not, but I was just looking up the authors of that paper to see what they're doing now. Very sadly, I found out that one of the authors, Emil Stefanov, died last year at the age of 26.

http://www.rememberingemil.org

Re: Ask HN: Algorithms for text fingerprinting?

#32
post #14

Figured I'd chime in here since I developed an algorithm recently that could be applied to this problem with some basic ML. Basically the first step would be shingling the text (choosing a sampling domain) and generating a MinHash struct (computationally cheap) which can then be used to find the "similarity" between sets, or, the "Jaccard Index." If you're clever about this, you can use HyperLogLogs to encode these M…

Could you describe in a little more detail what you mean by this sentence:

> "... use HyperLogLogs to encode these MinHash structs gaining a great deal of speed with a marginal error rate, all while allowing for arbitrary N-levels of intersection"

Thanks

Re: Ask HN: Algorithms for text fingerprinting?

#33
Here's a quick one:

1) tokenize each text into a different bag(set) of words.

2) Compute the Jaccard index[1] using the two sets.

Here's another

1) tokenize each text into a multi-bag(set) of words, keeping track of token frequency

2) keeping the token frequency, order the sets into lists

3) map the lists of words onto an n-dimensional space (where n is say...all of the words into the two documents) as vectors

4) compute the cosine similarity [2]

Here's another:

1) tokenize the texts into two bags of words

2) compute the set difference going both ways.

3) does either difference contain discriminator tokens that rule it out as being from that person

4 (optional)): extend to 2-3-n-grams

Here's another (a variant of the one above):

1) compute 1-2-3-n-grams from one of the texts

2) insert the n-grams into a set

3) compute the same for the second document and test for set membership

4) compute the number of total n-grams from your second document

5) compute (non-in-set/total-n-grams) * 100 to yield a "uniqueness" measure

6) determine if the second document is "unique" enough

And another:

1) assuming you have a sample corpus from a writer and want to know if a new text belongs in that corpus

2) follow the method above but for step #1 and 2 do it with the entire reference corpus

And another:

1) produce an ontology of discriminator terms and categories unique to the writer

2) use an (named entity recognition) NER tool of some kind to find those terms in each document

3) use the set of found terms as an alternative to a bag of words for the Jaccard or Vector models above

You may need to play with stopword list removal, tokenization schemes and n-gram windows (for example, omitting 1-grams might focus the analysis on phrase usage vs. vocabulary usage)

1 - https://en.wikipedia.org/wiki/Jaccard_index

2 - https://en.wikipedia.org/wiki/Cosine_similarity

Re: Ask HN: Algorithms for text fingerprinting?

#34

Earlier quoted context omitted.

Took only a minute to try: English -> Filipino (Tagalog) -> Chinese-simplified (Mandarin?) -> English I remember reading an article about a year ago (NSA) to identify the user, based on how they are written, vocabulary, spelling errors, grammar, language, and so on. It is interesting to me, because it is difficult to change the written and spoken word in use. It can be estimated that there are between two characters…

Of course you're sending all this information to Google now. Are there any offline translators that are advanced enough to be used for something like this? I imagine most just naively map words 1:1 which wouldn't do much good here.

I was wondering the same. Oddly enough, on your phone you can use Google Translate offline, they let you download language packs. Not sure if they ever "phone home".

Re: Ask HN: Algorithms for text fingerprinting?

#35
post #33

Here's a quick one: 1) tokenize each text into a different bag(set) of words. 2) Compute the Jaccard index[1] using the two sets. Here's another 1) tokenize each text into a multi-bag(set) of words, keeping track of token frequency 2) keeping the token frequency, order the sets into lists 3) map the lists of words onto an n-dimensional space (where n is say...all of the words into the two documents) as vectors 4) com…

This is an implementation in Python for the second option, but instead of word frequencies it uses word presence (a binary vector in the end):

https://github.com/atrilla/vsmpy

Re: Ask HN: Algorithms for text fingerprinting?

#36
post #9

The relevant search term is "stylometry". One particular paper I remember is from Dawn Song's group at Berkeley a couple years back: http://www.cs.berkeley.edu/~dawnsong/papers/2012%20On%20the%... There's a lot of public work on the topic, but it looks like right now the best place to look is still in academic papers (I don't know of any open source libraries, for example).

JGAAP [1] may be useful (AGPLv3 license):

"JGAAP is intended to tackle two different problems, firstly to allow people unfamiliar with machine learning and quantitative analysis the ability to use cutting edge techniques on their text based stylometry / textometry problems, and secondly to act as a framework for testing and comparing the effectiveness of different analytic techniques' performance on text analysis quickly and easily."

[1]: http://evllabs.com/jgaap/w/

Looks like there are some other recommendations at http://evllabs.com/jgaap/w/index.php/FAQ#What_other_tools_ar...

Re: Ask HN: Algorithms for text fingerprinting?

#38
post #14

Figured I'd chime in here since I developed an algorithm recently that could be applied to this problem with some basic ML. Basically the first step would be shingling the text (choosing a sampling domain) and generating a MinHash struct (computationally cheap) which can then be used to find the "similarity" between sets, or, the "Jaccard Index." If you're clever about this, you can use HyperLogLogs to encode these M…

Could you describe in a little more detail what you mean by this sentence: > "... use HyperLogLogs to encode these MinHash structs gaining a great deal of speed with a marginal error rate, all while allowing for arbitrary N-levels of intersection" Thanks

Sure thing.

Let's say we have two sets, and we're trying to find out how similar they are:

    setOne = ['the','brown','fox','jumped','a','log']
    setTwo = ['the','quick','brown','log','jumped','over','the','fox']
You could use an array intersection when its small, but if you want to do this efficiently at scale you need to take advantage of probabilistic data structures.

Let's say you created a HyperLogLog for each set:

    setOne = (bitfield representing setOne)
    setTwo = (bitfield representing setTwo)
Now HyperLogLogs are cool, because you can merge them together w/o losing anything. You can't retrieve the data, but you can efficiently check if a value exists inside. You can have a false positive, but never a false negative.

You might first try simple combinatorics (a la, similarity ~= |A or B| / |A ∪ B|) however this can get hairy depending upon the representation of the HyperLogLog (sparse/dense) and its respective cardinality.

Eventually you realize you can't accept an exponentially-compounding error rate, but still need the raw efficiency, and thus you sacrifice by doubling your storage cost.

Now instead of just the initial sets, you'd also build a parallel MinHash struct:

    setOne_minhashes = [, , ]
    setTwo_minhashes = [, , ]

    setOne = (bitfield representing setOne)
    setTwo = (bitfield representing setTwo)
    setOneMinHash = (bitfield representation of setOne minhashes)
    setTwoMinHash = (bitfield representation of setTwo minhashes)
I won't go into excessive detail about the minhash algorithm itself, but essentially it provides a way to sample a large set of values by selecting/retaining the smallest output hashes. What that means is, you can then intersect the minhash bitfields as many times over as you like, and extract a predictably accurate similarity index in linear time with definable confidence bounds.

Re: Ask HN: Algorithms for text fingerprinting?

#39
post #31
post #9

The relevant search term is "stylometry". One particular paper I remember is from Dawn Song's group at Berkeley a couple years back: http://www.cs.berkeley.edu/~dawnsong/papers/2012%20On%20the%... There's a lot of public work on the topic, but it looks like right now the best place to look is still in academic papers (I don't know of any open source libraries, for example).

I'm not sure whether it's appropriate to mention this or not, but I was just looking up the authors of that paper to see what they're doing now. Very sadly, I found out that one of the authors, Emil Stefanov, died last year at the age of 26. http://www.rememberingemil.org

Thank you for posting this (it was quite a surprise to see his name here).

I was a close friend of Emil for a long time. Before he passed, I had gotten busy with work and hadn't spoken to him in a while. I saw some of our mutual friends the weekend before it happened and had planned to call him. Really wish I had made that call sooner.

Re: Ask HN: Algorithms for text fingerprinting?

#40
post #38

Earlier quoted context omitted.

Could you describe in a little more detail what you mean by this sentence: > "... use HyperLogLogs to encode these MinHash structs gaining a great deal of speed with a marginal error rate, all while allowing for arbitrary N-levels of intersection" Thanks

Sure thing. Let's say we have two sets, and we're trying to find out how similar they are: setOne = ['the','brown','fox','jumped','a','log'] setTwo = ['the','quick','brown','log','jumped','over','the','fox'] You could use an array intersection when its small, but if you want to do this efficiently at scale you need to take advantage of probabilistic data structures. Let's say you created a HyperLogLog for each set: s…

I don't have a background in ML (I'm still somewhat early in my CS Bachelor's degree, and I'm not focusing on ML anyway), but this is very interesting, regardless that I had to look up nearly every term you used.

I was initially thinking a directed weighted graph might work well here, but I'm assuming that would scale terribly relative to something like this.

Post reply on HN