Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Text Mining Accessible to Any Developer & Non-Expert

blog.recursivity.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#21

Post by zeratul »

Text mining: most of the time is spent on gathering the data, curating the data, and working with your annotators (domain experts). After that, you try a dozen or more ways to covert documents into a matrix format. Then, you try a dozen or more feature selection algorithms. Finally, the icing on the cake: you get to try a dozen or more machine learning algorithms, each having a dozen or more parameters to be estimated.

Yep, it would be very nice to have an API that would do all that for you. But that would require a group of at least 10 ML experts + 10 NLP experts + 20 domain experts. Still, I think it's doable and one should make small efforts to make it happen.

Marginal thoughts: decision trees are very bad for large p >> n problems - random forest might work, though. If TextMinr doesn't have radial SVM with auto-tuning then it will not cope with more difficult problems.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#22

Post by St-Clock »

law wrote: post #18

I have a good amount of experience in natural language processing and machine learning, and I don't think offering an API that provides easy access to the algorithms is the right solution. The major algorithms in text classification aren't that complex to implement, and can be done in a few hundred lines. Moreover, all of the most widely used, widely tested, and reliable algorithms have public implementations that ar…

The other big catch is randomness, which is often not understood by neophytes. If you try to find relations in sufficiently large data set, you're bound to find some that are "caused" by randomness. Tools like p-values are of little help when you fish for many relations (and not just one in particular).

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#23

Post by wfaler »

zeratul wrote: post #21

Text mining: most of the time is spent on gathering the data, curating the data, and working with your annotators (domain experts). After that, you try a dozen or more ways to covert documents into a matrix format. Then, you try a dozen or more feature selection algorithms. Finally, the icing on the cake: you get to try a dozen or more machine learning algorithms, each having a dozen or more parameters to be estimate…

Appreciate your comment. And you're right about decision trees, though they can be useful for simpler problems, such as classifying documents into categories, whereby you have "sub categories" and the parent categories are mutually exclusive.

Decision trees are really only useful for problems where there is mutual exclusion between the different options, so they are definitely no silver bullet.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#24

Post by law »

St-Clock wrote: post #22
law wrote: post #18

I have a good amount of experience in natural language processing and machine learning, and I don't think offering an API that provides easy access to the algorithms is the right solution. The major algorithms in text classification aren't that complex to implement, and can be done in a few hundred lines. Moreover, all of the most widely used, widely tested, and reliable algorithms have public implementations that ar…

The other big catch is randomness, which is often not understood by neophytes. If you try to find relations in sufficiently large data set, you're bound to find some that are "caused" by randomness. Tools like p-values are of little help when you fish for many relations (and not just one in particular).

Randomness can be a curse, but can also be a blessing when introduced as in the random subspace methods. This again abstracts to understanding your business needs and whether the results encountered make sense given the features' [absence of] independence. An API giving you a wide choice of algorithms will still rely on you to run something like ICA as a pre-processing step to identify this statistically independent randomness.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#25

Post by law »

wfaler wrote: post #23
zeratul wrote: post #21

Text mining: most of the time is spent on gathering the data, curating the data, and working with your annotators (domain experts). After that, you try a dozen or more ways to covert documents into a matrix format. Then, you try a dozen or more feature selection algorithms. Finally, the icing on the cake: you get to try a dozen or more machine learning algorithms, each having a dozen or more parameters to be estimate…

Appreciate your comment. And you're right about decision trees, though they can be useful for simpler problems, such as classifying documents into categories, whereby you have "sub categories" and the parent categories are mutually exclusive. Decision trees are really only useful for problems where there is mutual exclusion between the different options, so they are definitely no silver bullet.

Fully grown decision trees are notorious for their risk of overfitting your training set. If you're uncomfortable fully growing the trees, you then have to consider whether you want to grow them out completely and then prune them, stop growing after a specific depth, train the trees using a random subset of features in the feature space (and then how many do you select? Do you use the square root? Logarithm?), etc. Even then, what are you using to choose when a node splits? Information gain? Information gain ratio? Gini index? What about when you have a feature like credit card numbers, which are unique?

These are all choices that the user has to make. For something as seemingly simple as a decision tree, you can see why some knowledge is required before embarking on any machine learning mission.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#26

Post by zeratul »

wfaler wrote: post #23
zeratul wrote: post #21

Text mining: most of the time is spent on gathering the data, curating the data, and working with your annotators (domain experts). After that, you try a dozen or more ways to covert documents into a matrix format. Then, you try a dozen or more feature selection algorithms. Finally, the icing on the cake: you get to try a dozen or more machine learning algorithms, each having a dozen or more parameters to be estimate…

Appreciate your comment. And you're right about decision trees, though they can be useful for simpler problems, such as classifying documents into categories, whereby you have "sub categories" and the parent categories are mutually exclusive. Decision trees are really only useful for problems where there is mutual exclusion between the different options, so they are definitely no silver bullet.

I suspect that you are referring to what's called "hierarchical text classification". If so then any classifier can be used for that. And it's not a simple problem. I found it to be a good way to deal with unbalanced classes if you understand the domain knowledge that sits behind your class labels. I suggest taking look at these papers:

http://scholar.google.com/scholar?q=%22hierarchical+text+cla...

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#27

Post by PaulHoule »

law wrote: post #18

I have a good amount of experience in natural language processing and machine learning, and I don't think offering an API that provides easy access to the algorithms is the right solution. The major algorithms in text classification aren't that complex to implement, and can be done in a few hundred lines. Moreover, all of the most widely used, widely tested, and reliable algorithms have public implementations that ar…

Well. Here's my take.

There are a number of text analysis SaaS offerings such as OpenCalais, AlchemyAPI, Zemanta, and OpenAmplify. They've all got impressive science under the hood, but none of them are accurate enough to be useful.

I spend most of my time these days thinking about why that is and what to do about it.

For systems to do better, they'll need to incorporate world knowledge; they'll need to test different interpretations of a text and select the ones that "make sense". This is likely to be a form of statistical inference rather than Cyc style logic.

Based on some systems I've worked with, I'd estimate that a space optimized "background" knowledge base that can estimate satisfiability in the common sense domain is on the order of 10-100 GB. It will puff out to at least an order of magnitude beyond that in the process of creating it.

Few users will have the ability to create a KB of that type, and it would be a serious thing to download and install.

Hosting the services of that kind of system in a SaaS manner makes a lot of sense.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#29

Post by zeratul »

PaulHoule wrote: post #27
law wrote: post #18

I have a good amount of experience in natural language processing and machine learning, and I don't think offering an API that provides easy access to the algorithms is the right solution. The major algorithms in text classification aren't that complex to implement, and can be done in a few hundred lines. Moreover, all of the most widely used, widely tested, and reliable algorithms have public implementations that ar…

Well. Here's my take. There are a number of text analysis SaaS offerings such as OpenCalais, AlchemyAPI, Zemanta, and OpenAmplify. They've all got impressive science under the hood, but none of them are accurate enough to be useful. I spend most of my time these days thinking about why that is and what to do about it. For systems to do better, they'll need to incorporate world knowledge; they'll need to test differen…

Without looking under the hood I'd say there could be at least four reasons why they fail (based on what most of the NLP literature is lacking):

- did not remove contradicting information from the training sets (two very similar vectors having contradicting labels)

- did not try enough feature selection algorithms

- did not estimate ALL learner parameters using the training sets with internal CV

- did not include domain knowledge

The last one refers to Paul Houle comment. Just, beside using tools like OpenCyc, WordNet, UMLS, there many other ways to embed domain expertise in an automated classification process. Injecting semantically related features into a vector representation of a document is extremely difficult. Forward feature selection doesn't work well for sparse and noisy data.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#30

Post by PaulHoule »

zeratul wrote: post #29
PaulHoule wrote: post #27

Earlier quoted context omitted.

Well. Here's my take. There are a number of text analysis SaaS offerings such as OpenCalais, AlchemyAPI, Zemanta, and OpenAmplify. They've all got impressive science under the hood, but none of them are accurate enough to be useful. I spend most of my time these days thinking about why that is and what to do about it. For systems to do better, they'll need to incorporate world knowledge; they'll need to test differen…

Without looking under the hood I'd say there could be at least four reasons why they fail (based on what most of the NLP literature is lacking): - did not remove contradicting information from the training sets (two very similar vectors having contradicting labels) - did not try enough feature selection algorithms - did not estimate ALL learner parameters using the training sets with internal CV - did not include dom…

the curse of dimensionality is the worst problem that affects machine learning

customers don't want to create training sets large enough to train text classifiers; often the number of documents they need to sort into a category is too small to fit in a category.

As for semantic indexing, it was hard to do in 2005. In 2011 it's easy. DBpedia and Freebase are a chromosome map for the human memome. With large amounts of instance information, it's possible to do things that a big rulebox can't.

These tools are aiming for the market segment that Cyc aimed for, but will use very different methodologies.

Post reply on HN