Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fitting an elephant with four non-zero parameters

arxiv.org

Post reply on HN
Print view View original

81–90 of 155 posts

Re: Fitting an elephant with four non-zero parameters

#81

Post by godelski »

nyssos wrote: post #74
soist wrote: post #66

Earlier quoted context omitted.

And I am saying they are confused because they are attributing personal characteristics to computers and software. By spelling out what computers are doing it becomes very obvious that there is nothing that can be aware of any experiences in computers as it is all simply a sequence of arithmetic operations. If you can explain which sequence of arithmetic operations corresponds to "experiences" in computers then you m…

> By spelling out what computers are doing it becomes very obvious that there is nothing that can be aware of any experiences in computers as it is all simply a sequence of arithmetic operations. By spelling out what brains are doing it becomes very obvious that it's all simply a sequence of chemical reactions - and yet here we are, having experiences. Software will never have a human experience - but neither will a…

If an algorithmic process is an experience and a collection of experiences is intelligence then we get some pretty wild conclusions that I don't think most people would be attempting to claim as it'd make them sound like a lunatic (or a hippy).

Consider the (algorithmic) mechanical process of screwing in a screw into a board. This screw has an "experience" and therefore intelligence. So... The screw is intelligent? Very low intelligence, but intelligent according to this definition.

But we have an even bigger problem. There's the metaset of experiences, that's the collection of several screws (or the screw, board, and screwdriver together). So we now have a meta intelligence! And we have several because there's the different operations on these sets to perform.

You might be okay with this or maybe you're saying it needs memory. If the later you hopefully quickly realize this means a classic computer is intelligent but due to the many ways information can be stored it does not solve our above conundrum.

So we must then come to the conclusion that all things AND any set of things have intelligence. Which kinda makes the whole discussion meaningless. Or, we must need a more refined definition of intelligence which more closely reflects what people actually are trying to convey when they use this word.

Re: Fitting an elephant with four non-zero parameters

#82

Post by godelski »

soist wrote: post #67
rocqua wrote: post #61

Earlier quoted context omitted.

Is over fitting different from 'too much optimization'? Optimization still needs a value that is optimized. Over fitting is the result of too much optimization for not quite the right value (i.e. training error when you want to reduce prediction error)

What value is being optimized and how do you know it is too much or not enough?

I think the miscommunication is due to the proxy nature of our modeling. From one perspective, yes you're right because it's just on your optimization function and objectives. But if we're in the context where we recognize the practical usage of our model replies on it being an inexact representation (proxy) then certainly there is too much optimization. I mean most of what we try to model in ML is intractable.

In fact, that entire notion of early stopping is due to this. We use a validation set as a pseudo test set to inject information into our optimization products without leaking information from the test set (why you shouldn't choose parameters based on test results. That is spoilage. Doesn't matter if it's status quo, it's spoilage)

But we also need to consider that a lack of divergence between train/val does not mean there isn't overfittng. Divergence implies overfittng but the inverse statement is not true. I state this because it's both relevant here and an extremely common mistake.

Re: Fitting an elephant with four non-zero parameters

#83

Post by andrewflnr »

karmakaze wrote: post #43
jampekka wrote: post #21

Earlier quoted context omitted.

This was mentioned in the first paragraph of the paper. The paper is mostly humoristic. That said, the wisdom of the quip has been widely lost in many fields. In many fields data is "modeled" with huge regression models with dozens of parameters or even neural networks with billions of parameters. > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elepha…

That's how I feel about dark matter. Oh this galaxy is slower than this other similar one. The first one must have less dark matter then. What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.

Dark matter is constrained by, among other things, dynamical simulations. For instance, here's an example of reproducing real world observations, that previously didn't have great explanations, using simulations with dark matter: https://www.youtube.com/live/8rok8E_tz8k?si=Q7vmQYpZr_6K7--m. And that's not even getting into the cosmology that has to (and mostly does) fit together.

Re: Fitting an elephant with four non-zero parameters

#84

Post by andrewflnr »

GuB-42 wrote: post #64
karmakaze wrote: post #43

Earlier quoted context omitted.

That's how I feel about dark matter. Oh this galaxy is slower than this other similar one. The first one must have less dark matter then. What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.

The dark matter theory is falsifiable. Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects, and it has to follow the laws of physics as we understand them today. It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interac…

> For the Occam razor, adding some random term to an equation is not really better than adding some invisible but well characterized stuff...

You're being too kind. It's worse. Especially when (in my understanding anyway) that added term doesn't even explain all the things dark matter does.

Re: Fitting an elephant with four non-zero parameters

#85

Post by soist »

godelski wrote: post #82
soist wrote: post #67

Earlier quoted context omitted.

What value is being optimized and how do you know it is too much or not enough?

I think the miscommunication is due to the proxy nature of our modeling. From one perspective, yes you're right because it's just on your optimization function and objectives. But if we're in the context where we recognize the practical usage of our model replies on it being an inexact representation (proxy) then certainly there is too much optimization. I mean most of what we try to model in ML is intractable. In fa…

Most practitioners seem to understand that what they are doing is creating executable models and they don't confuse the model based on numeric observations with the actual reality. This is why I very much do not like all the AI hype and how statistical models were rebranded as artificial "intelligence" because the people who are not aware of what the words mean get very confused and start thinking they are nothing more than computers executing algorithms to fit numerical data to some unspecified cognitive model.

Re: Fitting an elephant with four non-zero parameters

#86

Post by godelski »

soist wrote: post #85
godelski wrote: post #82

Earlier quoted context omitted.

I think the miscommunication is due to the proxy nature of our modeling. From one perspective, yes you're right because it's just on your optimization function and objectives. But if we're in the context where we recognize the practical usage of our model replies on it being an inexact representation (proxy) then certainly there is too much optimization. I mean most of what we try to model in ML is intractable. In fa…

Most practitioners seem to understand that what they are doing is creating executable models and they don't confuse the model based on numeric observations with the actual reality. This is why I very much do not like all the AI hype and how statistical models were rebranded as artificial "intelligence" because the people who are not aware of what the words mean get very confused and start thinking they are nothing mo…

> Most practitioners seem to understand that what they are doing is creating executable models and they don't confuse the model based on numeric observations with the actual reality.

I think you're being too optimistic, and I'm a pretty optimistic person. Maybe it is because I work in ML, but I've had to explain to a large number of people this concept. This doesn't matter if it is academia or industry. It is true for both management and coworkers. As far as I can tell, people seem very happy to operate under the assumption that benchmark results are strong indicators of real world performance __without__ the need to consider assumptions of your metrics or data. I've even proven this to a team at a trillion dollar company where I showed a model with lower test set performance had more than double the performance on actual customer data. Response was "cool, but we're training a much larger model on more data, so we're going to use that because it is a bit better than yours." My point was that the problem still exists in that bigger model with more data, but that increased params and data do a better job at hiding the underlying (and solvable!) issues.

In other words, in my experience people are happy to be Freeman Dyson in the conversation Calavar linked[0] and very upset to hear Fermi's critique: being able to fit data doesn't mean shit without either a clear model or a rigorous mathematical basis. Much of data science is happy to just curve fit. But why shouldn't they? You advance your career in the same way, by bureaucrats who understand the context of metrics even less.

I've just experienced too many people who cannot distinguish empirical results from causal models. And a lot of people who passionately insist there is no difference.

[0] https://news.ycombinator.com/item?id=40964328

Re: Fitting an elephant with four non-zero parameters

#87

Post by adrian_b »

GuB-42 wrote: post #64
karmakaze wrote: post #43

Earlier quoted context omitted.

That's how I feel about dark matter. Oh this galaxy is slower than this other similar one. The first one must have less dark matter then. What can't be fit by declaring the amount of dark matter that must be present fits the data? It's unfalsifiable, just because we haven't found it, doesn't mean it doesn't exist. Even worse than string/M-theory which at least has math.

The dark matter theory is falsifiable. Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects, and it has to follow the laws of physics as we understand them today. It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interac…

If we add an arbitrary amount of dark matter everywhere, to match the observed motions of the celestial bodies, that adds an infinity of parameters, and not even a enumerable one.

This obviously can match almost anything and it has extremely low predictive power (many future observations may differ from predictions, which can be accounted by some dark matter whose distribution was previously unknown), so it is a much worse explanation than a modified theory of gravity that would have only a finite number of additional parameters.

Re: Fitting an elephant with four non-zero parameters

#88

Post by adrian_b »

andrewflnr wrote: post #84
GuB-42 wrote: post #64

Earlier quoted context omitted.

The dark matter theory is falsifiable. Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects, and it has to follow the laws of physics as we understand them today. It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interac…

> For the Occam razor, adding some random term to an equation is not really better than adding some invisible but well characterized stuff... You're being too kind. It's worse. Especially when (in my understanding anyway) that added term doesn't even explain all the things dark matter does.

Adding any finite number of parameters is strictly better than adding an infinity of parameters (i.e. an arbitrary distribution of dark matter chosen to match the observations).

Re: Fitting an elephant with four non-zero parameters

#89

Post by smallnamespace »

adrian_b wrote: post #87
GuB-42 wrote: post #64

Earlier quoted context omitted.

The dark matter theory is falsifiable. Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects, and it has to follow the laws of physics as we understand them today. It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interac…

If we add an arbitrary amount of dark matter everywhere, to match the observed motions of the celestial bodies, that adds an infinity of parameters, and not even a enumerable one. This obviously can match almost anything and it has extremely low predictive power (many future observations may differ from predictions, which can be accounted by some dark matter whose distribution was previously unknown), so it is a much…

> to match the observed motions of the celestial bodies

The point is that even with current observational data there's no reasonable distribution of dark matter that correctly explains all evidence that we have.

Your intuition that "if I have an infinite number of degrees of freedom anything at all can be fit" is leading you astray here.

Re: Fitting an elephant with four non-zero parameters

#90

Post by adrian_b »

nyssos wrote: post #71
GuB-42 wrote: post #64

Earlier quoted context omitted.

The dark matter theory is falsifiable. Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects, and it has to follow the laws of physics as we understand them today. It is actually a satisfying theory with regard to the Occam razor. We don't have to change our laws of physics to explain the abnormal rotations of galaxy, we just need "stuff" that we can't see and yet interac…

> Sure we can't see dark matter (it doesn't interact electromagnetically), but we can see its effects Even this is granting too much: "seeing it" and "seeing its effects" are the same thing . No one has ever "directly seen", in the sense that internet DM skepticism demands, anything other than a photon.

"Seeing" is indeed a poorly chosen word.

The problem with dark matter is that there does not exist any second relationship from which to verify its existence, like in the case of normal matter, which takes part in a variety of interactions that lead to measurable effects, which can be compared.

The amount and the location of dark matter is computed from the gravitational forces that explain the observed movements of the bodies, but there are no additional relationships with any other data, which could corroborate the computed distribution of dark matter. That is what some people mean by "seeing".

Post reply on HN