Data-Visualization Firm’s New Software Autonomously Finds Abstract Connections

> The power of Ayasdi is its unique ability to automatically discover insights — regardless of complexity — without asking questions. Ayasdi’s customers can finally learn the answers to questions that they didn’t know to ask in the first place. Simply stated, Ayasdi is ‘digital serendipity’.”

It’s a bold statement, however by using algebraic topology Ayasdi has managed to totally remove the human element that goes into data mining — and, as such, all the human bias that goes with it.

---

Don't companies with massive datasets already do this in some way? Google's system for piecing together what people "really want" seems such a large, open-ended inquiry that it seems some automated insight discovery must be done.

Also, wouldn't a machine that generates automated insights require another layer of meta-analysis to be able to sift through which insights are actually usable? Using the example of sports statistics, you could generate an infinite number of trivial relations between players and plays and teams...it seems that at some point, a human with real domain knowledge has to go in and program a filter system, which seems to be about the same amount of work as the inquiry-generation that this software automates.

Finally, why all the emphasis on visualization? Visualization helps to illustrate to humans the possibilities of investigation...for a machine that can supposedly ask (and answer) all the correct questions, isn't visualization merely eye candy?

It'd be great to see a concrete example of this software in action. Perhaps feed it the NFL play-by-play data that was posted on HN a few weeks ago and see if it can generate usable strategy.

It's a bit of an odd article, but my guess is that it's a gloss of a press release, and that kind of writing is par for the course for AI/ML press releases.

Gunnar Carlsson, the researcher mentioned, seems to work mainly on an approach related to manifold learning (roughly, finding lower-dimensional structure in high-dimensional data) that's based on algebraic topology. He co-ran a workshop on that last year at NIPS, one of the main machine-learning conferences: https://sites.google.com/site/nips2012topology/ . He's written some highly cited papers on the subject, though it'd be a bit of a stretch to claim he invented the area, since there have been workshops at least back to 2007, that one organized by a set of French researchers: http://topolearnnips2007.insa-rouen.fr/description.html

I would guess the part about removing the human element from data mining is putting an optimistic PR spin on the basic idea of automatically extracting lower-dimensional structure, which, if it works, should allow for less feature engineering. The emphasis on visualization makes sense in that light, if they're planning to sell it as a no-expertise-necessary system: feed it data and get interpretable-by-non-experts viz as output, with any complexity that would normally require "data scientists" being handled automatically.

You could probably already do that with the R package caret.

It can train models for you automatically, so you could loop over every model available and return the top ten predictions. Of course it wouldn't get around the need for feature engineering, but its theoretically possible.

Note - do not attempt this unless you have lots and lots of machines, as it will take a very, very long time.

So far what I see is an "innovative" service/tool with a non-innovative old (e.g. biotech) software business plan. I have to jump through hoops to get my hands on this tool. Have a phone call with them, talk about your data etc. Then, what is the point of easy UI and no-human-needed touch? Not to mention I have no idea about their price model.

I don't understand what is their cost in letting people try it before using the tool; unless of course it runs on their machines. Put it somewhere and let me try it with some simple data (e.g. iris in R).

Automatically finding connections in your data is easy. The problem is finding connections that are 1) non-obvious and 2) actionable. And when they are found, filtered so that they stand out from all the obvious, non-actionable connections.

In other words, if I'm trying to find out why a plane crashed, and the first thing this system tells me is gravity, then it isn't a practical advancement.