Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billion-Parameter Theories

worldgov.org

Post reply on HN
Print view View original

31–40 of 92 posts

Re: Billion-Parameter Theories

#31

Post by cwmoore »

suddenlybananas wrote: post #9
curuinor wrote: post #6

Connectionist models have lots of theory by theoreticians explicitly pissed off about Chomsky's assertion that there is an inbuilt ability for language. Jay McClelland's office had a little corkboard thingy with Chomsky mockery on the side, for example. Putting forth even the implicature that the present direct descendants are intellectual descendants of Chomsky is like saying Protestants are intellectual descendants…

>Jay McClelland's office had a little corkboard thingy with Chomsky mockery on the side, for example. I've never understood why the idea of linguistic nativism is so upsetting to people.

Indeed, operating human lips, teeth, tongue, and larynx is far beyond language models.

Re: Billion-Parameter Theories

#32

Post by zkmon »

> It's remarkable how much of reality turned out to be modelable by theories that fit in a few symbols.

The admiration for "remarkable" things puts humanity on a dangerous path that is disconnected from the real goals of human progress as a species. You don't need any of this compression of knowledge or truths. Folklore tales about celestial bodies are fine and hood enough. The vulgar pursuit for knowledge is paving the way for extinction of humans as biological creatures.

Re: Billion-Parameter Theories

#33

Post by bbor »

suddenlybananas wrote: post #9
curuinor wrote: post #6

Connectionist models have lots of theory by theoreticians explicitly pissed off about Chomsky's assertion that there is an inbuilt ability for language. Jay McClelland's office had a little corkboard thingy with Chomsky mockery on the side, for example. Putting forth even the implicature that the present direct descendants are intellectual descendants of Chomsky is like saying Protestants are intellectual descendants…

>Jay McClelland's office had a little corkboard thingy with Chomsky mockery on the side, for example. I've never understood why the idea of linguistic nativism is so upsetting to people.

Well that anecdote is referencing the Scruffies v. Neat war[1], within which the nativism debate was merely a somewhat-archaic undercurrent.

IMHO, a lot of the more specifically anti-nativist sentiments of today are based in linguistics itself rather than philosophy, CS, or CogSci, where again it is part of a broader (and much dumber) debate: whether linguistics is the empirical study of languages or the theoretical study of language itself. People get really nasty when they're told that they work in an offshoot field for some reason, which is why I blame them for the ever-too-common misunderstandings of Chomsky -- the most common being "Universal Grammar has been disproven because babies don't speak English in the womb".

If Chomsky weren't so obviously right, this would be a worrying development! Luckily I expect it to be little more than a footnote in history, so it's merely infuriating rather than depressing.

[1] Minsky, 1991: https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...

Re: Billion-Parameter Theories

#34

Post by aldousd666 »

harperlee wrote: post #5

Two handwavey ideas upon reading this: - Even for billion-parameter theories, a small amount of vectors might dominate the behaviour. A coordinate shift approach (PCA) might surface new concepts that enable us to model that phenomenon. "A change in perspective is worth 80 IQ points", said Alan Kay. - There is analogue of how we come up with cognitive metaphors of the mind ("our models of the mind resemble our latest…

I don't disagree, but neither does the article. It's just talking about the fact that we previously considered anything that can't be easily and tersely written down as nearly or entirely intractable. But, as we have seen, the three body problem is not really a hum-dinger as far as the universe goes, it's not even table stakes. We need to be able to do the same kind of energy arbitrage on n-body problems that we do on 2. And now we have the beginnings of a place to toy with more complicated ideas -- since these won't fit on a blackboard.

Re: Billion-Parameter Theories

#35

Post by bbor »

cwmoore wrote: post #31
suddenlybananas wrote: post #9

Earlier quoted context omitted.

>Jay McClelland's office had a little corkboard thingy with Chomsky mockery on the side, for example. I've never understood why the idea of linguistic nativism is so upsetting to people.

Indeed, operating human lips, teeth, tongue, and larynx is far beyond language models.

Apologies if I'm stepping on a joke, but just in case: Nativism is about cognitive capacities, not sensorimotor ones. All apes could easily communicate just as well as Helen Keller, yet none of them have ever asked a question, much less written a book!

Re: Billion-Parameter Theories

#36

Post by rbanffy »

If we think of spacetime as some sort of cellular automaton, where each state of a given point is a function (with some randomness, because God likes to throw dice) of previous states of the surrounding points, if the rules for a new state generation are extremely complex, there will be some significant overhead in dimensions we don't see, because the rules need to be somehow represented outside the observable reality. Another issue with this idea is that while the rules might be "outside", the parameters themselves have to be somehow encoded in the state of a cell, and can't propagate faster than light, or one cell (an indivisible unit of space) per indivisible unit of time), which limits the number of parameters accessible to any given cell to the ones immediately surrounding it.

Disclaimer: I hope it's obvious, but I'm no physicist. This is just how I would build a universe.

Re: Billion-Parameter Theories

#37

Post by pixl97 »

aldousd666 wrote: post #34
harperlee wrote: post #5

Two handwavey ideas upon reading this: - Even for billion-parameter theories, a small amount of vectors might dominate the behaviour. A coordinate shift approach (PCA) might surface new concepts that enable us to model that phenomenon. "A change in perspective is worth 80 IQ points", said Alan Kay. - There is analogue of how we come up with cognitive metaphors of the mind ("our models of the mind resemble our latest…

I don't disagree, but neither does the article. It's just talking about the fact that we previously considered anything that can't be easily and tersely written down as nearly or entirely intractable. But, as we have seen, the three body problem is not really a hum-dinger as far as the universe goes, it's not even table stakes. We need to be able to do the same kind of energy arbitrage on n-body problems that we do o…

Problems with opaque stability boundaries that observe non-liner effects are always great. Chaos theory makes it even more fun as your observation can change the outcome.

Re: Billion-Parameter Theories

#39

Post by pixl97 »

zkmon wrote: post #32

> It's remarkable how much of reality turned out to be modelable by theories that fit in a few symbols. The admiration for "remarkable" things puts humanity on a dangerous path that is disconnected from the real goals of human progress as a species. You don't need any of this compression of knowledge or truths. Folklore tales about celestial bodies are fine and hood enough. The vulgar pursuit for knowledge is paving…

Right, dinosaurs were perfectly fine, their ignorance worked out well for them.

The universe is uncaring, simply not giving a shit if you have knowledge or not. Knowledge gives you the ability to survive minor conniption fits of cosmic magnitude, and at the same time gives you a gun to shoot your own foot off.

There ain't no such thing as a free lunch.

Re: Billion-Parameter Theories

#40

Post by js8 »

munificent wrote: post #29
js8 wrote: post #4

I disagree with the article. I think it is always possible to come up with reasonably small theories that capture most of the given phenomena. So in a sense, you don't need complex theories in the form of large NNs (models? functions? programs?), other than for more precise prediction. For example - global warming. It's nice to have AOGCMs that have everything and the carbon sink in them. But if you want to understan…

> I think it is always possible to come up with reasonably small theories that capture most of the given phenomena. I can write a program (call it a simulation of some artificial phenomenon) whose internal logic is arbitrarily complex. The result is irreducible: the entire byzantine program with all of its convoluted logic is the smallest possible theory to describe the phenomenon, and yet the theory is not reasonabl…

That's true but I can still approximate what the system does with a simpler model. For example, I can split states of the system into n distinct groups, and measure transition probabilities between them.

Thermodynamics is a classic example of a phenomenological model like that.

Post reply on HN