Live data from Hacker News

The Size and Shape of “Idea Space” (2011) [pdf]

scaledinnovation.com

1–5 of 5 posts

Re: The Size and Shape of “Idea Space” (2011) [pdf]

#2
Fascinating!

The methods are very interesting -- a threshold (similar to "just noticeable difference" (JND) used in cognitive/perceptual science); an attempt at estimating the size of the space; an attempt using Hausdorff dimension to approximate the number of dimensions (one could consider a multi-dimensional scaling approach to this too).

But the choice of data and features means the results shouldn't be taken too seriously:

> We have implemented such a capability using feature counting, where a feature can be a word or phrase with statistically significant frequency, or the author’s name, or specific text from popup-menu selections, or if available, background database demographics about the author (his or her department, location, title, etc.).

If you use e.g. author's name and demographics as features, then this will contaminate the features that describe the ideas themselves. The rest of the results just aren't worth reading about until that problem is fixed.

Re: The Size and Shape of “Idea Space” (2011) [pdf]

#4

I wonder if the dimensionality 14 is something intrinsic to idea spaces in general, or if they're actually measuring the dimensionality of the human brain's storage of ideas? Or something about the English language even?

For me, any such number is an artefact of the data -- they've included things like the name and demographics of the idea's author, which will be clearly useful in practice for a "Show me more ideas like this" application, but which will only be misleading if trying to study the nature of "idea space".