Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billion-Parameter Theories

worldgov.org

Post reply on HN
Print view View original

21–30 of 92 posts

Re: Billion-Parameter Theories

#21

Post by delichon »

b450 wrote: post #12

Reminds me of the blog post about Waymo's "World Model". Training on real-world data results in a sufficiently rich model to start simulating novel scenarios that aren't in the training data (like the elephant wandering into the street), which in turn can feed back into training. One could imagine scientific inquiry working the same way. It strikes me that many of these complex systems have indeterminate boundaries,…

> like the elephant wandering into the street

Or a dinosaur that looks like it might:

https://x.com/phatman_19/status/2030728278437491102

Re: Billion-Parameter Theories

#22

Post by galaxyLogic »

ileonichwiesz wrote: post #10

This might be an unkind reading, but to me this just sounds like an attempt to reinvent the very same kind of mysticism that it mentions in the first paragraph. “No need to study the world around you and wonder about its rules, peasant - it’s far beyond your understanding! Only ~the gods~ computers can ever know the truth!” I shudder to think about a future where people give up on working to understand complex system…

Mark Cubain had a good line, I don't know if he came up with it or who, but he reportedly said:

" There are 2 types of people using AI: Those who use it so they can know everything, and those who use it so they don't have to know anything. " :-

Re: Billion-Parameter Theories

#23

Post by simianwords »

harperlee wrote: post #5

Two handwavey ideas upon reading this: - Even for billion-parameter theories, a small amount of vectors might dominate the behaviour. A coordinate shift approach (PCA) might surface new concepts that enable us to model that phenomenon. "A change in perspective is worth 80 IQ points", said Alan Kay. - There is analogue of how we come up with cognitive metaphors of the mind ("our models of the mind resemble our latest…

Maybe we can come up with smaller models that perform almost as well as the bigger ones. Could that just be pca of some kind?

Gpt nano vs gpt 5 for example.

Re: Billion-Parameter Theories

#24

Post by bee_rider »

dakiol wrote: post #8

> You could capture the behavior of every falling object on Earth in three variables and describe the relationship between matter and energy in five characters. What we can do is to approximate. Newton had a good approximation some time ago about gravitation (force equals a constant times two masses divided by distance squared. Super readable indeed) But nowadays there's a better one that doesn't look like Newton's t…

Einstein’s equations look like Newton’s in the limit. It would be a little weird if we ended up having to add millions of additional parameters over the next thousand years. At the current rate we seem to get multiple years per parameter, rather than hundreds of parameters per year, right?

Re: Billion-Parameter Theories

#25

Post by pdonis »

js8 wrote: post #4

I disagree with the article. I think it is always possible to come up with reasonably small theories that capture most of the given phenomena. So in a sense, you don't need complex theories in the form of large NNs (models? functions? programs?), other than for more precise prediction. For example - global warming. It's nice to have AOGCMs that have everything and the carbon sink in them. But if you want to understan…

> I also don't think poverty is a complex problem, but that's a minor point.

I'm not sure it's a minor point. I don't think poverty is a "complex" problem either, as that term is used in the article, but that doesn't mean I think it fits into one of the other two categories in the article. I think it is in a fourth category that the article doesn't even consider.

For lack of a better term, I'll call that category "political". The key thing with this category of problems is that they are about fundamental conflicts of interest and values, and that's a different kind of problem from the kind the article talks about. We don't have poverty in the world because we lack accurate enough knowledge of how to create the wealth that brings people out of poverty. We have poverty in the world because there are people in positions of power all over the world who literally don't care about ending poverty, and who subvert attempts to do so--who make a living by stealing wealth instead of creating it, and don't care that that means making lots of other people poor.

Re: Billion-Parameter Theories

#26

Post by xikrib »

Let's gather authors of 15 different world languages together in a room and see if they can collaboratively write a short story. Surely their inability to do so will prove their inadequacy in their native language. /s

Simplicity brings us closer to truth — Occam's razor has underpinned the development of our species for centuries. It's enterprise, empire, and capital that feed off of complexity.

We're entering a period of human history where engineers and businesspeople drive academic discourse, rather than scientists or philosophers. The result is intellectual chicken scratch like this article.

Re: Billion-Parameter Theories

#28

Post by MarkusQ »

niemandhier wrote: post #7

He talks about the Santa Fe institute and how they failed to carry their findings into the real world. They did not. They showed that for certain problems one could not do more than figure out some invariant and scaling laws. Showing what is impossible is not failure. For the rest: Modern gene networks and lots of biological modelling is based on their work as well as quite a few other things. That’s also not failure…

Clarke's second law:

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

Also see Minsky's "Perceptrons"

The problem with almost all such proofs is that people (even those who know better) read them as "this can't be done" when in fact they tell you "it can't be done unless you break one of the following assumptions."

I agree that it's unfair to say they failed, but it's likewise unfair to say that their success was in telling us our limits rather than exploring what we need to do to get around the roadblocks.

Re: Billion-Parameter Theories

#29

Post by munificent »

js8 wrote: post #4

I disagree with the article. I think it is always possible to come up with reasonably small theories that capture most of the given phenomena. So in a sense, you don't need complex theories in the form of large NNs (models? functions? programs?), other than for more precise prediction. For example - global warming. It's nice to have AOGCMs that have everything and the carbon sink in them. But if you want to understan…

> I think it is always possible to come up with reasonably small theories that capture most of the given phenomena.

I can write a program (call it a simulation of some artificial phenomenon) whose internal logic is arbitrarily complex. The result is irreducible: the entire byzantine program with all of its convoluted logic is the smallest possible theory to describe the phenomenon, and yet the theory is not reasonably small for any reasonable definition.

Re: Billion-Parameter Theories

#30

Post by bbor » 

  There's a parallel in linguistics. Chomsky showed that all human languages share deep recursive structure. True, and essentially irrelevant to the language modeling that actually learned to do something with language.
...this is so absurdly and blatantly wrong that it's hard to move past. Has the author ever heard of programming languages??
Post reply on HN