Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fitting an elephant with four non-zero parameters

arxiv.org

Post reply on HN
Print view View original

61–70 of 155 posts

Re: Fitting an elephant with four non-zero parameters

#61

Post by rocqua »

soist wrote: post #19
luplex wrote: post #17

Earlier quoted context omitted.

I mean the devil is in the details. In Reinforcement Learning, the target moves! In deep learning, you often do things like early stopping to prevent too much optimization.

There is no such thing as too much optimization. Early stopping is to prevent overfitting to the training set. It's a trick just like most advances in deep learning because the underlying mathematics is fundamentally not suited for creating intelligent agents.

Is over fitting different from 'too much optimization'? Optimization still needs a value that is optimized. Over fitting is the result of too much optimization for not quite the right value (i.e. training error when you want to reduce prediction error)

Re: Fitting an elephant with four non-zero parameters

#62

Post by jampekka »

edflsafoiewq wrote: post #50
karmakaze wrote: post #43

Earlier quoted context omitted.

That's how I feel about dark matter. Oh this galaxy is slower than this other similar one. The first one must have less dark matter then. What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.

It's easy to say "Epicycles! Epicycles!", but people are going to continue using their epicycles until a Copernicus comes along.

There will be no Copernicus if everybody just studies epicycles. E.g. there are massive resources put into the desperate WIMP hunt that could be used for finding new theories.

Re: Fitting an elephant with four non-zero parameters

#63

Post by rocqua »

jampekka wrote: post #58
rocqua wrote: post #56

Earlier quoted context omitted.

I used to think this, but dark matter does make useful predictions, that are hard to explain otherwise. This is partially because there are two ways to detect dark-matter. The first is gravitational lensing. The second is the rotatinal speed of galaxies. There are some galaxies that need less Dark Matter to explain their rotational speed. We can then cross check whether those galaxies cause less gravitational lensing…

The problem with dark matter is that there's no (working) theory on how the dark matter is distributed. It's really easy to "explain" gravitational effects if you can postulate extra mass ad-hoc to fit the observations.

If there are two different types of observations, and one parameter can explain both, that is pretty strong evidence. Put differently, dark matter is falsifyable, and experiments have tried to falsify it without success.

Besides the idea 'not all mass can be seen optically' is not that surprising. The many theories on what that mass might be are all speculation, but they are treated as such.

Re: Fitting an elephant with four non-zero parameters

#64

Post by GuB-42 »

karmakaze wrote: post #43
jampekka wrote: post #21

Earlier quoted context omitted.

This was mentioned in the first paragraph of the paper. The paper is mostly humoristic. That said, the wisdom of the quip has been widely lost in many fields. In many fields data is "modeled" with huge regression models with dozens of parameters or even neural networks with billions of parameters. > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elepha…

That's how I feel about dark matter. Oh this galaxy is slower than this other similar one. The first one must have less dark matter then. What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.

The dark matter theory is falsifiable. Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects, and it has to follow the laws of physics as we understand them today.

It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interact gravitationally. When we have stuff like neutrinos, it is not that far fetched. In fact, though unlikely given our current understanding of physics, dark matter could be neutrinos.

If, as it turn out, the invisible stuff we call dark matter doesn't follow the laws of physics as we know them, then the dark matter theory is falsified and we need a new one (or at least some tweaks). And it may actually be the case as a recent paper claims that gravitational lensing doesn't match the predictions of the dark matter theory.

The main competitor to dark matter is modified gravity, which calls for no new stuff, but changes the equations for gravity. For the Occam razor, adding some random term to an equation is not really better than adding some invisible but well characterized stuff, especially when we consider that the equation in question is extremely well tested. It is, of course, also falsifiable.

The problem right now is not that these theories are unfalsifiable, it is that they are already pretty much falsified in their current form (dark matter less than modified gravity), and some rework is needed.

Re: Fitting an elephant with four non-zero parameters

#65

Post by soist »

rocqua wrote: post #60
soist wrote: post #18

Earlier quoted context omitted.

AlphaZero did not create any experiences. AlphaZero was software written by people to play board games and that's all it ever did.

Are you launching into a semantic argument about the word 'experience'? If so, it might help to state what essential properties alphago was missing that makes it 'not having an experience'. Otherwise this can quickly devolve into the common useless semantic discussion.

Just making sure no one is confused by common computationalist sophistry and how they attribute personal characteristics to computers and software. People can have and can create experiences, computers can only execute their programmed instructions.

Re: Fitting an elephant with four non-zero parameters

#66

Post by soist »

pharrington wrote: post #47
soist wrote: post #42

Earlier quoted context omitted.

I am familiar with the literature on reinforcement learning.

They're saying the board games AlphaZero played with itself are experiences.

And I am saying they are confused because they are attributing personal characteristics to computers and software. By spelling out what computers are doing it becomes very obvious that there is nothing that can be aware of any experiences in computers as it is all simply a sequence of arithmetic operations. If you can explain which sequence of arithmetic operations corresponds to "experiences" in computers then you might be less confused than all the people who keep claiming computers can think and feel.

Re: Fitting an elephant with four non-zero parameters

#67

Post by soist »

rocqua wrote: post #61
soist wrote: post #19

Earlier quoted context omitted.

There is no such thing as too much optimization. Early stopping is to prevent overfitting to the training set. It's a trick just like most advances in deep learning because the underlying mathematics is fundamentally not suited for creating intelligent agents.

Is over fitting different from 'too much optimization'? Optimization still needs a value that is optimized. Over fitting is the result of too much optimization for not quite the right value (i.e. training error when you want to reduce prediction error)

What value is being optimized and how do you know it is too much or not enough?

Re: Fitting an elephant with four non-zero parameters

#68

Post by Calavar »

jampekka wrote: post #21
maitola wrote: post #16

I love the ironic side of the article. Perhaps they should add the reason for it, from Fermi's and Neumann's. When you are building a model of reality in Physics, If something doesn’t fit the experiments, you can’t just add a parameter (or more) variate it and fit the data. The model should have zero parameters, ideally, or the least possible, or, even at a more deeper level, the parameters should emerge naturally fr…

This was mentioned in the first paragraph of the paper. The paper is mostly humoristic. That said, the wisdom of the quip has been widely lost in many fields. In many fields data is "modeled" with huge regression models with dozens of parameters or even neural networks with billions of parameters. > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elepha…

> > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.”

For those who are interested, you can watch Freeman Dyson recount this conversation in his own words in an interview: https://youtu.be/hV41QEKiMlM

Re: Fitting an elephant with four non-zero parameters

#69

Post by nyssos »

karmakaze wrote: post #43
jampekka wrote: post #21

Earlier quoted context omitted.

This was mentioned in the first paragraph of the paper. The paper is mostly humoristic. That said, the wisdom of the quip has been widely lost in many fields. In many fields data is "modeled" with huge regression models with dozens of parameters or even neural networks with billions of parameters. > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elepha…

That's how I feel about dark matter. Oh this galaxy is slower than this other similar one. The first one must have less dark matter then. What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.

> What can't be fit by declaring the amount of dark matter that must be present fits the data?

Tons of things - just like there are tons of things that can't be fit by declaring the amount of electromagnetically-interacting matter that must be present fits the data.

You can fit anything you like by positing new and more complicated laws of physics, but that's not what's going on here. Dark matter is ordinary mass gravitating in an ordinary way: the observed gravitational lensing needs to match up with the rotation curves needs to match up with the velocity distributions of galaxies in clusters; you don't strictly need large scale homogeneity and isotropy but you really really want it, etc. Lambda-CDM doesn't handle everything perfectly (which in itself demonstrates that it's not mindless overfitting) but neither does anything else.

Re: Fitting an elephant with four non-zero parameters

#70

Post by HeatrayEnjoyer »

soist wrote: post #65
rocqua wrote: post #60

Earlier quoted context omitted.

Are you launching into a semantic argument about the word 'experience'? If so, it might help to state what essential properties alphago was missing that makes it 'not having an experience'. Otherwise this can quickly devolve into the common useless semantic discussion.

Just making sure no one is confused by common computationalist sophistry and how they attribute personal characteristics to computers and software. People can have and can create experiences, computers can only execute their programmed instructions.

On what priors are you making that statement?
Post reply on HN