11–20 of 29 posts
years ago i got my hands wet with some rudimentary text classification and grouping techniques. it was when google news came out, and it was fluffier than i like my news to be (it has since been tuned a lot better). so i wrote my own.
i wrote up my (admittedly naive and simplistic - but effective) methods here:
http://www.informit.com/articles/printerfriendly.aspx?p=3988...
Earlier quoted context omitted.
You have to manually mark entities though, how is this advantageous over say indexing your data using solr or whatever, and using even the built-in clustering tools? (Carrot by default)
I didn't realize solr had clustering. One difference might be the ability to pass in scoring, include, and dynamic tagging functions at clustering time. Does carrot cluster on arbitrary document fields?
We focus solely on sports and our classifiers(supervised) reduce the scope first to the sport and then to the specific team before we apply any sort of clustering (k-means, LDA, etc). That allows us to reduce the vocabulary to what is mostly a list of named entities for the sport/team and key words such as 'injury', 'quarterback', etc. With a significantly reduced vocabulary, even algorithms such as Hierarchical LDA work surprisingly well.
how are you doing entity extraction?
1. scalability: does your system ingest multiple documents in parallel? if so, how often do you observe over-segmentation, if any?
2. thresholds: how did you set the thresholds at various parts of the systems?
The underlying dilemma is that so many of these stores are not really "related". They're just the same story, rewritten off of a press release. The ideal system would pick out clusters, but also have sub-clusters within the cluster that would contain articles on the same subject but with diverse info.
nice article. a few questions here 1. scalability: does your system ingest multiple documents in parallel? if so, how often do you observe over-segmentation, if any? 2. thresholds: how did you set the thresholds at various parts of the systems?