I'm trying to automate the classification of documents that we are seeing from our crawler. I have done a google search for keyword classification using Python but I'm not getting any joy. Anyone here automatically tagging or classifying documents (Maybe after some teaching) ? Would be good to know how if you are
Ask YC: How would you do keyword classification
1–10 of 21 posts
Re: Ask YC: How would you do keyword classification
#2i think you are looking for document classification algorithms: http://en.wikipedia.org/wiki/Document_classification
the current state of the art algorithms are based on support vector machines, but their learning part could be tricky to implement in a scalable fashion. if you are looking for a quick and dirty approach, TFIDF algorithm (it is a naive "naive Bayes" :) is simple and is adequate for many applications
Re: Ask YC: How would you do keyword classification
#3i think you are looking for document classification algorithms: http://en.wikipedia.org/wiki/Document_classification the current state of the art algorithms are based on support vector machines, but their learning part could be tricky to implement in a scalable fashion. if you are looking for a quick and dirty approach, TFIDF algorithm (it is a naive "naive Bayes" :) is simple and is adequate for many applications
forgot to mention that you may want to look at the Orange framework which is in Python
http://www.ailab.si/orange/
Re: Ask YC: How would you do keyword classification
#4i think you are looking for document classification algorithms: http://en.wikipedia.org/wiki/Document_classification the current state of the art algorithms are based on support vector machines, but their learning part could be tricky to implement in a scalable fashion. if you are looking for a quick and dirty approach, TFIDF algorithm (it is a naive "naive Bayes" :) is simple and is adequate for many applications
forgot to mention that you may want to look at the Orange framework which is in Python http://www.ailab.si/orange/
great! that does look interesting. I'll have a good look through it. You involved in this field yourself? If so, drop me a line as we're looking for a consultant to help us with something
Re: Ask YC: How would you do keyword classification
#5Earlier quoted context omitted.
forgot to mention that you may want to look at the Orange framework which is in Python http://www.ailab.si/orange/
great! that does look interesting. I'll have a good look through it. You involved in this field yourself? If so, drop me a line as we're looking for a consultant to help us with something
I've done some work on machine learning and specifically document classification in a corporate behemoth. Send me an email to gmail.com and prefix that with osipov followed by an @ sign -- I'll get back to you.
Re: Ask YC: How would you do keyword classification
#6 I use the Yahoo Term Extractor API. This probably doesn't meet your long term needs, but it's great for prototyping.
http://developer.yahoo.com/search/content/V1/termExtraction....
Re: Ask YC: How would you do keyword classification
#7you may want to try, or match your results against:
Re: Ask YC: How would you do keyword classification
#8LSI - Latent Semantic Analysis is pretty useful in this field. Python has an amazing toolkit for it
Email me if you have any questions-I've been playing with this stuff for a while and it's really interesting.
Re: Ask YC: How would you do keyword classification
#9I use the Yahoo Term Extractor API. This probably doesn't meet your long term needs, but it's great for prototyping. http://developer.yahoo.com/search/content/V1/termExtraction....
Just note that you are limited to 5k queries per 24 hours per IP address with this. Like Tony said, NOT for production.
Re: Ask YC: How would you do keyword classification
#10I'm playing with some code for this just for fun. There are two ways to classify a document:
1. You already have a set of keywords (categories) that the document can belong to. The objective is to match a document to its category. This is true classification.
2. You want to extract relevant keywords from a document. This is not classification in the true sense but keyword extraction.
Each one has different approaches to achieve it, but they are similar problems. (As an example of similarity: you have a set of categories each defined by a tag cloud. You extract keywords from a document and see which tag cloud it matches best.)
So how do you do each one?
Classification: I'm not well versed in this area and I'm interested in learning - it's next on my to-do list.
Keyword Extraction: Yahoo! has an API to do that, but honestly, it's rubbish. I don't know how it "works" but it doesn't really. Open Calais is really good but has a noticeable error rate (I didn't quantify it but after trying many documents with it, I regularly noticed minor mistakes).
Hope this helps.