Live data from Hacker News

Fitting an elephant with four non-zero parameters

arxiv.org

31–40 of 155 posts

Re: Fitting an elephant with four non-zero parameters

#31
post #24
post #16

I love the ironic side of the article. Perhaps they should add the reason for it, from Fermi's and Neumann's. When you are building a model of reality in Physics, If something doesn’t fit the experiments, you can’t just add a parameter (or more) variate it and fit the data. The model should have zero parameters, ideally, or the least possible, or, even at a more deeper level, the parameters should emerge naturally fr…

Hmm.. Hodgin and Huxley did ground-breaking work on squid's giant axon and modelled neural activity. They had multiple parameters extracted from 'curve fitting' of recorded potential and injected currents which were much later mapped to sodium channels. Similarly, another process to potassium channels. I woudnt worry too much having multiple parameters -- even four when 3 can't just explain the model.

Neuron anatomy is the product of hundreds of millions of years of brute contingency. There are reasons why it can't be certain ways (organisms that were that way [would have] died or failed to reproduce) but no reason whatsoever why it had to be exactly this way. It didn't, there are plenty of other ways that nerves could have worked, this is just the way they actually do.

The physics equivalent is something like eternal inflation as an explanation for apparent fine-tuning - except that even if it's correct it's still absolutely nowhere near as complex or as contingent as biology.

Re: Fitting an elephant with four non-zero parameters

#32
post #20

This is humorous (and well-written), but I think its more than that. I'm always making the joke (observation) that ML (AI) is just curve-fitting. Whether "just curve-fitting" is enough to produce something "intelligent" is, IMO, currently unanswered, largely due to differing viewpoints on the meaning of "intelligent". In this case they're demonstrating some very clean, easy-to-understand curve-fitting, but it's reall…

In the case of AI, the more parameters, the better! In Physics is the opposite.

One of the hardest parts of training models is avoiding overfitting, so "more parameters are better" should be more like "more parameters are better given you're using those parameters in the right way, which can get hard and complicated".

Also LLMs just straight up do overfit, which makes them function as a database, but a really bad one. So while more parameters might just be better, that feels like a cop-out to the real problem. TBD what scaling issues we hit in the future.

Re: Fitting an elephant with four non-zero parameters

#33
post #16

I love the ironic side of the article. Perhaps they should add the reason for it, from Fermi's and Neumann's. When you are building a model of reality in Physics, If something doesn’t fit the experiments, you can’t just add a parameter (or more) variate it and fit the data. The model should have zero parameters, ideally, or the least possible, or, even at a more deeper level, the parameters should emerge naturally fr…

The balance between empirical data fitting and genuine understanding of the underlying reality

Re: Fitting an elephant with four non-zero parameters

#34
> It only satisfies a weaker condition, i.e., using four non-zero parameters instead of four parameters.

Why would that be a harder problem? In the case that you get a zero parameter, you could inflate it by some epsilon and the solution would basically be the same.

Re: Fitting an elephant with four non-zero parameters

#35

This is humorous (and well-written), but I think its more than that. I'm always making the joke (observation) that ML (AI) is just curve-fitting. Whether "just curve-fitting" is enough to produce something "intelligent" is, IMO, currently unanswered, largely due to differing viewpoints on the meaning of "intelligent". In this case they're demonstrating some very clean, easy-to-understand curve-fitting, but it's reall…

Your humorous observation captures a fundamental truth to some extent

Re: Fitting an elephant with four non-zero parameters

#36
post #20

This is humorous (and well-written), but I think its more than that. I'm always making the joke (observation) that ML (AI) is just curve-fitting. Whether "just curve-fitting" is enough to produce something "intelligent" is, IMO, currently unanswered, largely due to differing viewpoints on the meaning of "intelligent". In this case they're demonstrating some very clean, easy-to-understand curve-fitting, but it's reall…

In the case of AI, the more parameters, the better! In Physics is the opposite.

A dichotomy between these fields

Re: Fitting an elephant with four non-zero parameters

#37
post #4

I wish there was more humor on arXiv. If I could make a discovery in my own time without using company resources I would absolutely publish it in the most humorous way possible.

Pretraining on the Test Set Is All You Need

https://arxiv.org/abs/2309.08632

Re: Fitting an elephant with four non-zero parameters

#40
post #18
post #15

Earlier quoted context omitted.

> Whether "just curve fitting" is enough to produce something "intelligent" is, IMO, currently unanswered Continual "curve fitting" to the real world can create intelligence. What is missing is not something inside the model. It's missing a mechanism to explore, search and expand its experience. Our current crop of LLMs ride on human experience, they have not largely participated in creating their own experiences. Th…

AlphaZero did not create any experiences. AlphaZero was software written by people to play board games and that's all it ever did.

AZ trained in self-play mode for millions of games, over multiple generations of a player pool.
Post reply on HN