Ask HN: Is artificial intelligence/natural language processing a futile pursuit?

Hi. I am a sophomore with a start-up that builds sentiment analysis technologies. We currently work on sentiment summarization and classification. Using papers and patents as a starting point, we try to create our own algorithm that we hope will outperform state-of-the-art approaches. We have been working on it for the past 2 months. Call it an irrational mindset, but we're adamant on building something that is not only tedious to copy but also fundamentally hard to duplicate.

We have concerns regarding what we are doing, namely: technology and market risks.

1) Technology risk We are worried that we would hit a dead-end and not be able to build what we set out to build. AI is a hard technical problem, and Marc Andreessen has described AI as an "equivalent" of rocket science. We are making progress but our chances of technologically outperforming Google or other AI-based companies are inherently not good. We fear that we might have picked the wrong beast to mess with but we're a little too involved right now to switch paths.

2) Market risk Other than social media monitoring tools, I haven't come across any solution that deals with sentiment analytics. The social media monitoring scene is crowded, so we hope to apply what we build in other unexplored areas. Our vision for now is to tackle the problem of information overload within consumer review platforms: think Amazon reviews, Yelp reviews, IMDb reviews, etc. We hope to offer an on-site (within-the-page-itself) dashboard that categorises raw data by mentioned concepts and their frequencies, the sentiments (positive/negative) with regard to said concepts and selected quotes that incorporate said aspects, along with sentiment insights in visualised forms (e.g. pie charts). End-users can interact with the dashboard and use any variable as an anchor to obtain insights (e.g. give me the raw data that mention concepts 1 and 2 in a positive light). This, we hope, would help people consume content more representatively and faster, without having to read through every single review from top-left to bottom-right. Hopefully, this would increase user engagement and shorten sales cycles. Naturally, the fear is that nobody wants our product, much less pay for it. Personally, as an end-user, I would find something like that useful, but I'm obviously biased.

Looking forward to some input for my current situation.

We got rocket science pretty much right 50 years ago. Don't let the fact that it sounds hard discourage you. Somebody's going to get this right eventually - why not you?

There have been a great many successes in the AI field. It's easy for people to forget though; once something works, nobody calls it AI anymore.

Hi danielh,

Yes, there are already advertising networks that do semantic-targeted contextual advertising, e.g. peer39.com, collective.com. Not sure how they are doing though. Either way, it is something that we will look into. Thanks.

Don't worry about the technology risk. People already do this, so it isn't impossible. They probably don't do it well, but you only have to do better than random to provide benefit.

Market risk is more of a concern to me. If you're presenting results directly to users they'll probably care greatly about the quality of your algorithm. If you're just using aggregated data to drive, for example, marketing campaigns then the noise will be washed out given enough data.

I'd start by trying to validate your business model.

1) You need to find out who the paying customers are in this space and what features they really want. It seems very possible that average consumers would have no interest in your service but that power users, marketers or some other segment might. Once you know where the interest is you can make better feature decisions. For example, you might find that reporting, not AI, is what drives marketing sales.

2) Fundemental improvements in AI are really hard. Look at the results of the first Netflix prize. Lots of world class teams worked on that problem and the winning solution only produced something like a 10% improvement in the results. If that's the only edge your business has over the competition I doubt consumers would even notice. On the other hand, AI has matured to the point where it's pretty easy to produce good results. I'd rather bet my business on predicatble, good results and treat it as a pleasant surprise if we make a fundemental breakthrough.

Good luck!