Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Counterintuitive Properties of High Dimensional Space

marckhoury.github.io

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Counterintuitive Properties of High Dimensional Space

#61

Post by OscarCunningham »

algorias wrote: post #59
OscarCunningham wrote: post #56

Earlier quoted context omitted.

The convention is to name things for their inherent dimensionalitly, not the space they live in. A sphere in three dimensional space is a two dimensional object, in the sense that it's a surface on which the points can be described by two coordinates (say longitude and latitude). Similarly a point on a circle can be described by a single coordinate (distance around the circle). So the 1-sphere is a circle, and normal…

If you talk about the "volume of a sphere" (like the article does), then a sphere embedded in 3 dimensions should be 3 dimensional. This would usually be called a ball and not a sphere, though. Note that analogously, a cube in 3 dimensional space is always considered to be 3 dimensional.

Right, the 2-sphere is the surface of the 3-ball. I think "volume of a sphere" is okay though. It just means the volume enclosed.

>a cube in 3 dimensional space is always considered to be 3 dimensional.

So "cubes" are taken to include their interior. That makes sense I suppose. I wonder what the surface of a cube is called. A "box" maybe?

Re: Counterintuitive Properties of High Dimensional Space

#62

Post by nerdponx »

btilly wrote: post #13

Anyone interested in practical consequences of these counterintuitive properties should look up https://en.wikipedia.org/wiki/Curse_of_dimensionality for how it impacts things like machine learning where we naturally are working with many dimensions.

The behavior the article is, to me, way more bizarre than the curse of dimensionality.

It's tempting to think of data sets as "point clouds". This article is a reality check for me: you can't safely apply intuition about 2- and 3-d point clouds to higher dimensional data. I suspect that this explains why methods like tSNE seem to produce unstable results depending on the parameters [0]. The notion of a "neighbor" in high dimensions is just not what I think it is.

I suppose the same is true for high-dimensional cost surfaces. Gradient descent is often described as "like walking down a hill". But without a deep understanding of high-dimensional geometry, I'm not at all confident that I know what a 4-, 10-, or 1000-dimensional hill looks like.

The lesson: Be skeptical of my own geometric intuition unless it is firmly backed by math.

[0]: https://distill.pub/2016/misread-tsne/

Re: Counterintuitive Properties of High Dimensional Space

#63

Post by HelloNurse »

OscarCunningham wrote: post #56
coinerone wrote: post #53

"You may be used to using the word “circle” in two dimensions and “sphere” in three dimensions. However, in higher dimensions we generally just use the word sphere, or d-sphere when the dimension of the sphere is not clear from context. With this terminology, a circle is also called a 1-sphere, for a 1-dimensional sphere. A standard sphere in three dimensions is called a 2-sphere, and so on. " So, what is a 2-dimensi…

The convention is to name things for their inherent dimensionalitly, not the space they live in. A sphere in three dimensional space is a two dimensional object, in the sense that it's a surface on which the points can be described by two coordinates (say longitude and latitude). Similarly a point on a circle can be described by a single coordinate (distance around the circle). So the 1-sphere is a circle, and normal…

Things that are named for their inherent dimensionality regardless of the dimensionality of the space they are contained in are others, such as points (0), lines (1), planes (2), general manifolds, vector spaces, etc. Spheres (more properly balls) and cubes (or hypercubes) are understood to have the same number of dimensions as the space they are contained in.

Re: Counterintuitive Properties of High Dimensional Space

#64

Post by btilly »

nerdponx wrote: post #62
btilly wrote: post #13

Anyone interested in practical consequences of these counterintuitive properties should look up https://en.wikipedia.org/wiki/Curse_of_dimensionality for how it impacts things like machine learning where we naturally are working with many dimensions.

The behavior the article is, to me, way more bizarre than the curse of dimensionality. It's tempting to think of data sets as "point clouds". This article is a reality check for me: you can't safely apply intuition about 2- and 3-d point clouds to higher dimensional data. I suspect that this explains why methods like tSNE seem to produce unstable results depending on the parameters [0]. The notion of a "neighbor" in…

Indeed. I find https://www.thestar.com/news/insight/2016/01/16/when-us-air-... to be a good cautionary tale on how our intuition about people being close to average is misleading - nobody is. And nobody is particularly like anyone else, either.

On a thousand dimensional hill, my intuition is that it locally looks like a low dimensional hill, along axes that you can find through techniques like Principal Components Analysis. This has yet to mislead me. On the other hand, my pure math background was a long time ago, and I have not explored machine learning in any real depth...

Re: Counterintuitive Properties of High Dimensional Space

#65

Post by afpx »

visarga wrote: post #50

An interesting property of word vectors (which are usually 300-600 dimensional vectors) is that most are quasi-orthogonal. That means when you sum them up, they compose well representing all the meanings of the component parts, and strangely, multi-sense words such as "bank" contain all the senses overlapped, yet distinct. Another interesting property is that high dimensional space has many shortcuts, or that at any…

Could this be one of the reasons why there are so many conspiracy theories these days? That is, with so many dimensions of data available, one can find close connections between any two things?

Re: Counterintuitive Properties of High Dimensional Space

#66

Post by mywittyname »

visarga wrote: post #50

An interesting property of word vectors (which are usually 300-600 dimensional vectors) is that most are quasi-orthogonal. That means when you sum them up, they compose well representing all the meanings of the component parts, and strangely, multi-sense words such as "bank" contain all the senses overlapped, yet distinct. Another interesting property is that high dimensional space has many shortcuts, or that at any…

> An interesting property of word vectors (which are usually 300-600 dimensional vectors) is that most are quasi-orthogonal.

I think it would be more interesting if this wasn't the case. The set of all possible English words is I'm assuming you mean that, "I will run with Bob" and "I will jog with Stacy" are not orthogonal, because they convey a very similar message, but are orthogonal to, "Man, that was a good beer."

Re: Counterintuitive Properties of High Dimensional Space

#67

Post by mywittyname »

afpx wrote: post #65
visarga wrote: post #50

An interesting property of word vectors (which are usually 300-600 dimensional vectors) is that most are quasi-orthogonal. That means when you sum them up, they compose well representing all the meanings of the component parts, and strangely, multi-sense words such as "bank" contain all the senses overlapped, yet distinct. Another interesting property is that high dimensional space has many shortcuts, or that at any…

Could this be one of the reasons why there are so many conspiracy theories these days? That is, with so many dimensions of data available, one can find close connections between any two things?

A conspiracy theorist will see a lack of evidence to the contrary as evidence of the affirmative. I.e., aliens exist because government officials let us in to the military base so we can see for ourselves there are no aliens there.

Whereas a reasonable person needs evidence of affirmative to believe something. I.e., news that NASA probes have discovered bacteria in subterranean wells on Mars.

The former way of thinking can be used to "prove" just about anything. So yeah, in some sense, most data points helps fuel more conspiracies because certain people are conditioned to believe anything.

Re: Counterintuitive Properties of High Dimensional Space

#68

Post by nerdponx »

btilly wrote: post #64
nerdponx wrote: post #62

Earlier quoted context omitted.

The behavior the article is, to me, way more bizarre than the curse of dimensionality. It's tempting to think of data sets as "point clouds". This article is a reality check for me: you can't safely apply intuition about 2- and 3-d point clouds to higher dimensional data. I suspect that this explains why methods like tSNE seem to produce unstable results depending on the parameters [0]. The notion of a "neighbor" in…

Indeed. I find https://www.thestar.com/news/insight/2016/01/16/when-us-air-... to be a good cautionary tale on how our intuition about people being close to average is misleading - nobody is. And nobody is particularly like anyone else, either. On a thousand dimensional hill, my intuition is that it locally looks like a low dimensional hill, along axes that you can find through techniques like Principal Components An…

But what this article makes me think is that even "low" dimensions can't be trusted, as long as it's greater than 3.
Post reply on HN