Big Data and the Topologist (2012)
ldtopology.wordpress.com
Big Data and the Topologist (2012)
1–10 of 22 posts
Re: Big Data and the Topologist (2012)
#2I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Re: Big Data and the Topologist (2012)
#3I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Out of curiosity what criteria did you use to indicate a homology element was persistent?
Re: Big Data and the Topologist (2012)
#4KXEN does some interesting things with assuming that what the data approximates is (I think) the solution set to a quadratic equation, rather than a linear one.
Re: Big Data and the Topologist (2012)
#5I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Well, the thing about noise, from a topological point of view, is that persistent homology simply cannot decide whether something is noise or not --- at least not right away.
To be more precise: A lot of what PH does is actually some sort of multi-scale Betti number calculation. The Betti numbers count the number of k-dimensional "holes" in a data set. Their calculation is usually done by something that is called "simplicial homology" in algebraic topology. Topologists like simplicial homology because it "only" requires that your input data is given in the form of a simplicial complex. A simplicial complex is a sort of generalized graph. Think of your input data is being described by vertices, edges, triangles, tetrahedra, and their higher-dimensional counterparts. Got it? Good.
Now, the problem about real-world data is that it does not come in the form of a simplicial complex. So, this sort of structure needs to be approximated somehow. And it is precisely through this approximation that the noise begins to creep in. PH tries to deal with the noise by assigning a weight to each feature it detects, sort of like a "scale" on which the feature "lives" (note that I am talking about a feature in the sense of a "hole" here). Features that live "long" are considered important. Features that don't live long, are not considered important.
So, coming finally back to your comment: You, as a user of PH, have to (sort of) decide what to consider as noise and what not. If you take a look at the seminal applications in PH, you will find that the extraction of features works quite well, although some sort of preprocessing might be required.
Anyway, if you are interested about using PH to analyse some data, I would be happy to discuss some stuff with you :)
Disclaimer: I am working with PH from the point of visualization. I try to make the topological structure of a data set visible.
Re: Big Data and the Topologist (2012)
#6I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Well, the thing about noise, from a topological point of view, is that persistent homology simply cannot decide whether something is noise or not --- at least not right away. To be more precise: A lot of what PH does is actually some sort of multi-scale Betti number calculation. The Betti numbers count the number of k-dimensional "holes" in a data set. Their calculation is usually done by something that is called "si…
Well said points. I've been digging into writing some data viz tools that leverage ph recently, and everything you say is exactly true.
Re: Big Data and the Topologist (2012)
#7I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Just a few days ago the following paper was published on the ArXiv, it specifically tackles the problem of noise. I haven't had time to read it thoroughly, but it also contains a nice introduction to persistent homology for statisticians, who are usually not familiar with the concepts of algebraic topology.
Re: Big Data and the Topologist (2012)
#8I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Just a few days ago the following paper was published on the ArXiv, it specifically tackles the problem of noise. I haven't had time to read it thoroughly, but it also contains a nice introduction to persistent homology for statisticians, who are usually not familiar with the concepts of algebraic topology. http://arxiv.org/pdf/1303.7117v1.pdf
A note -- when linking to arXiv, please link to the abstract, not directly to the PDF.
Re: Big Data and the Topologist (2012)
#9I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Well, the thing about noise, from a topological point of view, is that persistent homology simply cannot decide whether something is noise or not --- at least not right away. To be more precise: A lot of what PH does is actually some sort of multi-scale Betti number calculation. The Betti numbers count the number of k-dimensional "holes" in a data set. Their calculation is usually done by something that is called "si…
I am just starting to dig into PH for data analysis. I would be very interested in speaking with you more about the topic.
Re: Big Data and the Topologist (2012)
#10I tried analysing data using persistent homology. What is not obvious, although they do admit it in one line of every paper, is that is it susceptible to noise :( So it has to go in the bin even though I really want to know what my manifolds look like!
Just a few days ago the following paper was published on the ArXiv, it specifically tackles the problem of noise. I haven't had time to read it thoroughly, but it also contains a nice introduction to persistent homology for statisticians, who are usually not familiar with the concepts of algebraic topology. http://arxiv.org/pdf/1303.7117v1.pdf
Been working through the exposition in that paper the past few days, very very accessible writing. Very concrete intelligible examples. You can even ignore the few pieces or category theory and still take away a lot of understanding