Clustering related stories
blog.getprismatic.com
Clustering related stories
1–10 of 29 posts
Re: Clustering related stories
#2Since you have feature vectors, have you looked at using LSH to reduce the number of comparisons/memory consumption?
Re: Clustering related stories
#3Since you have feature vectors, have you looked at using LSH to reduce the number of comparisons/memory consumption?
In general, LSH is great for detecting near-duplicates which are very close, but we have two issues: (1) The stories we're looking for have a greater distance than LSH is great at detecting efficiently and (2) Because our feature vector has structure (named entities, n-grams, etc.) we have problem-specific 'coarse' feature-vector, which is good at detecting possible related candidates cheaply.
Re: Clustering related stories
#4The underlying dilemma is that so many of these stores are not really "related". They're just the same story, rewritten off of a press release. The ideal system would pick out clusters, but also have sub-clusters within the cluster that would contain articles on the same subject but with diverse info.
Re: Clustering related stories
#5Did you consider weighting of the different feature types (on top of tf/idf)?
Re: Clustering related stories
#6Did you consider weighting of the different feature types (on top of tf/idf)?
We actually do weight them differently, but the post was already pretty long so I had to gloss over some of the details.
Re: Clustering related stories
#7The underlying dilemma is that so many of these stores are not really "related". They're just the same story, rewritten off of a press release. The ideal system would pick out clusters, but also have sub-clusters within the cluster that would contain articles on the same subject but with diverse info.
Doesn't Google do something similar to this for their news aggregator?
Re: Clustering related stories
#8You might find this library helpful in this area -> https://github.com/rwynn/rugroupy.
Re: Clustering related stories
#9You might find this library helpful in this area -> https://github.com/rwynn/rugroupy .
You have to manually mark entities though, how is this advantageous over say indexing your data using solr or whatever, and using even the built-in clustering tools? (Carrot by default)
Re: Clustering related stories
#10You might find this library helpful in this area -> https://github.com/rwynn/rugroupy .
You have to manually mark entities though, how is this advantageous over say indexing your data using solr or whatever, and using even the built-in clustering tools? (Carrot by default)
I didn't realize solr had clustering. One difference might be the ability to pass in scoring, include, and dynamic tagging functions at clustering time. Does carrot cluster on arbitrary document fields?