Live data from Hacker News

Making Text Mining Accessible to Any Developer & Non-Expert

blog.recursivity.com

11–20 of 33 posts

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#11
I've found in my data-mining experience that the most interesting data (at least on the Web) is not particularly easy to parse, even if you write something that automates a form's POST submissions. The second difficult part is normalizing it, as much web/text data is formatted for display to humans, which is quite different than data in easily analyzable form.

So given that, it's just worth learning enough program to do loops, conditionals, and regexes to get what you want.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#12
post #8

Earlier quoted context omitted.

self driving cars autonomous helicopters automated analysis of satellite imagery search engines visual search engine (google goggles) virtual assistant (siri) speech recognition document classification (spam detectors) question answering systems (ibm watson) ad placement (google adsense) computer guided surgery high throughput imaging (chemo/bioinformatics) product recommendation (amazon/netflix) acturial science ind…

Nice! I think I should look into this. I sense that I lack a general understanding of what is really possible with ML. (My current understanding is weighted toward underestimating what is possible with ML).

That's a very good list. I think we're just starting to scratch at the surface of what is possible, which is why I've devoted myself to working in this field!

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#13
Text mining is one of my specialties and I have had similar ideas for a business. One thing that has stopped me is the awesome (and free for about 50K API calls a day) Open Calais service that does entity extraction and identifies some relationships between entities in input text.

For document clustering there are many good open source tools that people and companies can use. The commercial Ling Pipe product does a good job at sentiment analysis.

Obtaining, scrubbing, and generally curating the data is a pain point that users of this system may still need to worry about.

I wish this new business good luck, but there are definitely some real problems to work around. Perhaps we should go into business together :-)

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#14
post #10
post #4

Machine learning seems to be a meme on the rise right now. What kind of startups are possible in that domain?

Machine learning is concerned with developing algorithms that enable programs to use data to evolve new behavior. Calling it a meme seems to put it in the same category as advice animals and Old Spice ads. Still, I see what you're saying--- it is really "in" right now, which is probably because: (1) There's a lot more data now for people to use it with (2) Infrastructure for doing it is cheap and scalable (3) Automat…

Yeah, that's exactly what I meant.

I think the online ML class (Stanford?) is helping to pump up interest in the laycrowd.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#15
post #4

Machine learning seems to be a meme on the rise right now. What kind of startups are possible in that domain?

self driving cars autonomous helicopters automated analysis of satellite imagery search engines visual search engine (google goggles) virtual assistant (siri) speech recognition document classification (spam detectors) question answering systems (ibm watson) ad placement (google adsense) computer guided surgery high throughput imaging (chemo/bioinformatics) product recommendation (amazon/netflix) acturial science ind…

I got most of these right away, but could you expand on what you mean by "high throughput imaging (chemo/bioinformatics)"?

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#16
post #9

This probably won't work, as google found out with it's prediction api (it hasn't been used much). There's already enough open source software out there that's state of the art and easy to use. There's good business to be had in selling data though which is where these folks should probably divert their effort.

I don't know; ML has a lot of applications and the bar for most people to be able to implement it is rather high. Lowering the bar so "mere mortals" can have some serious infrastructure and data that's a mere API call away seems pretty huge.

The problem is that if you're not confident enough to get these systems working yourself, you're probably not going to be confident enough in your business to pay by the sip for someone else's api.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#17
post #15

Earlier quoted context omitted.

self driving cars autonomous helicopters automated analysis of satellite imagery search engines visual search engine (google goggles) virtual assistant (siri) speech recognition document classification (spam detectors) question answering systems (ibm watson) ad placement (google adsense) computer guided surgery high throughput imaging (chemo/bioinformatics) product recommendation (amazon/netflix) acturial science ind…

I got most of these right away, but could you expand on what you mean by "high throughput imaging (chemo/bioinformatics)"?

robots chugging away at petri dishes that contain hundreds of mini-dishes with chemicals/biostuff. Making sense of the millions/billions of data points is the job of machine learning.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#18
I have a good amount of experience in natural language processing and machine learning, and I don't think offering an API that provides easy access to the algorithms is the right solution. The major algorithms in text classification aren't that complex to implement, and can be done in a few hundred lines. Moreover, all of the most widely used, widely tested, and reliable algorithms have public implementations that are readily adaptable to your needs. And that's the problem: understanding your needs.

Understanding your needs (or your company's needs) is where people with PhDs make their money. Machine learning isn't a panacea, and we won't be seeing a one-size-fits-all approach for awhile. Even though data has become more accessible, it might be noisy, incomplete, streaming, partially labeled, etc. This is why understanding exactly what you're trying to model with these algorithms is crucial and why "just applying" them is impractical at best and misleading at worst.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#19
post #15

Earlier quoted context omitted.

self driving cars autonomous helicopters automated analysis of satellite imagery search engines visual search engine (google goggles) virtual assistant (siri) speech recognition document classification (spam detectors) question answering systems (ibm watson) ad placement (google adsense) computer guided surgery high throughput imaging (chemo/bioinformatics) product recommendation (amazon/netflix) acturial science ind…

I got most of these right away, but could you expand on what you mean by "high throughput imaging (chemo/bioinformatics)"?

The high throughput imaging that I'm familiar with is in regards to cell imaging. Cells are cultured in high-density plates (96-well or 384-well dishes), each well is given a different experimental condition (drugs, RNAi, etc) and then imaged on an automated microscope.

As you can imagine, this generates tons of data. Our lab did a highthrouput screen of genetic mutants in neurons, and then used software to quantify basic morphology such as neurite length , arborization, and cell death.

Crystallographers will use a similar system to bathe their protein in billions of compounds to find the right combination for crystallizing. Automated cameras will capture images and try to identify which ones have crystallized so the researcher doesn't have to do it by hand.

Re: Making Text Mining Accessible to Any Developer & Non-Expert

#20
post #11

I've found in my data-mining experience that the most interesting data (at least on the Web) is not particularly easy to parse, even if you write something that automates a form's POST submissions. The second difficult part is normalizing it, as much web/text data is formatted for display to humans, which is quite different than data in easily analyzable form. So given that, it's just worth learning enough program to…

This is so true. I've been mining mailing lists and framework documentation for my Ph.D. and most of my effort and time was spent normalizing the data. Once that was done, classifying content and linking concepts was relatively easy...
Post reply on HN