What corpus did you get your term frequencies from?
Clustering related stories
21–29 of 29 posts
Re: Clustering related stories
#22The underlying dilemma is that so many of these stores are not really "related". They're just the same story, rewritten off of a press release. The ideal system would pick out clusters, but also have sub-clusters within the cluster that would contain articles on the same subject but with diverse info.
Doesn't Google do something similar to this for their news aggregator?
They also have access to the link graph, so they probably use that for clustering instead of looking at the text. Pages with lots of inbound linkers in common are likely to be similar.
Re: Clustering related stories
#23What corpus did you get your term frequencies from?
If I understand the blog correctly, their corpus consists of all the articles they have processed so far. Maybe they have some additional source, e.g. a more general collection of web pages.
Re: Clustering related stories
#24I worked on something similar for clustering BBC news articles. The (Ruby) code I used is here: https://github.com/bbcrd/similarity
I didn't account for names entities or n-grams in the feature vector though. That's a very interesting idea.
@mattdeboard - what algorithm did you use to count the occurrence and size of clusters?
Re: Clustering related stories
#25I worked on something similar for my final-year project at University in Python, at https://github.com/basicallydan/Onda - clustering related articles in the online media.
In retrospect it's pretty ugly, but it worked pretty well. I really wanted to implement named entities and n-grams but never got around to it. I'm glad you guys did :)
Re: Clustering related stories
#26The underlying dilemma is that so many of these stores are not really "related". They're just the same story, rewritten off of a press release. The ideal system would pick out clusters, but also have sub-clusters within the cluster that would contain articles on the same subject but with diverse info.
It is helpful that the same story is often written off of PA or AP articles and thus includes pretty much the same info - but it doesn't change the fact that stories on the same subject almost always include the same set of key words unique to that subject whether or not they were rewritten from a press release. That's the beauty of TF.IDF weighting - that it'll cluster stuff based off of words that are uniquely important in one article.
Re: Clustering related stories
#27nice article. a few questions here 1. scalability: does your system ingest multiple documents in parallel? if so, how often do you observe over-segmentation, if any? 2. thresholds: how did you set the thresholds at various parts of the systems?
1. We do ingest multiple documents in parallel, but we also have a system in place for merging clusters, so it ends up not being a problem. 2. Pure hackery. Tweak, look at data, repeat.
Can you elaborate on how you merge clusters? Many times a clusters will be just single-doc clusters, how to check if you need to merge two single-doc clusters? (it seems like the same process that was done in the first place to create the clusters: doc x doc similarity)
Re: Clustering related stories
#28Earlier quoted context omitted.
1. We do ingest multiple documents in parallel, but we also have a system in place for merging clusters, so it ends up not being a problem. 2. Pure hackery. Tweak, look at data, repeat.
Can you elaborate on how you merge clusters? Many times a clusters will be just single-doc clusters, how to check if you need to merge two single-doc clusters? (it seems like the same process that was done in the first place to create the clusters: doc x doc similarity)
Merging is pretty simple, and could probably use a little more TLC. The way we do it is that when we get a new document in the system, if the similarity score is above some threshold for documents in two different clusters we will consider merging those clusters. We then make the yes/no decision by comparing random documents from both clusters and averaging the scores, but the threshold we use here is a bit lower than the non-merging decisions (since we have the additional information of this new document doing a good job of linking the clusters).
Re: Clustering related stories
#29Earlier quoted context omitted.
Can you elaborate on how you merge clusters? Many times a clusters will be just single-doc clusters, how to check if you need to merge two single-doc clusters? (it seems like the same process that was done in the first place to create the clusters: doc x doc similarity)
Merging is pretty simple, and could probably use a little more TLC. The way we do it is that when we get a new document in the system, if the similarity score is above some threshold for documents in two different clusters we will consider merging those clusters. We then make the yes/no decision by comparing random documents from both clusters and averaging the scores, but the threshold we use here is a bit lower tha…
this doesn't work in the case where there are two (and only two) similar documents get ingested into the system as new singleton clusters at the same time; this case is very rare so it is not a big issue to you, i guess.