Ask HN: Is artificial intelligence/natural language processing a futile pursuit?

Personally I find I need sophisticated domain-specific heuristics to evaluate consumer reviews in particular spaces. For instance, when I buy a lens for my camera, I'm going to look at reviews, but it's tricky because there are always good and bad reviews for any lens. Some of the people who get bad reviews had a camera with a screwed up AF or they never really understood how to use the lens or what its limitations were. Then you look at say, a Sigma lens for the Canon platform and you'll see that different people are having wildly different results and you'd better just forget about it.

Dashboards and stuff like that is a waste of time. What I really want is something that creates a super-expert opinion that's maybe 3 sentences to a paragraph wrong. A bit beyond the state of the art.

----

More generally, I think Doug Lenat had the right idea with Cyc, but he went about it the wrong way. Had Doug not been able to make a comfortable living doing work for the government, he would have been forced to produce a revolutionary product, but he wasn't.

I think that the linked data space around the semantic web is going to explode and ultimately produce the "commonsense" knowledgebase that it takes to build real NLP systems.

Most of the people I know in the knowledge-management space are trying to develop expensive projects for government, pharma, legal discovery and such. I think, however, they are in the "pay a lot, get a little" businesses that are going to be disrupted by the next wave. On one hand you've got Google, Microsoft and a few biggies that are going to develop large-scale but low-margin products. The other side is going to be a vast, largely low-margin market of operators who pull from and add to the great linked data pool... which is going to grow like a Katamari ball until we reach the Singularity, maybe around 2025 or so.

In my experience, the complexity of most NLP applications work against them in the sense that they're competing with simpler approaches that are less computationally intensive. Good technology will let you do things other people can't do, but if you want to make it a business you'll need to know how to compete against Google doing 80% of what you can do with simple pattern matching. Your results will need to be orders of magnitude better before you have something people will use, let alone for which they'll pay. And be careful not to get trapped in a field where you'll be competing against companies paying serious cash for commercial databases to which you do not have access.

I'm personally skeptical there is much of a market for sentiment analysis incidentally, but the same tools are pretty useful for search (preprocessing, etc.). I think you'll find it difficult to get third-party adoption unless your product drives direct revenue for someone or can very visibly improve their product. But the problems are important and worth solving!