Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billion-Parameter Theories

worldgov.org

Post reply on HN
Print view View original

11–20 of 92 posts

Re: Billion-Parameter Theories

#11

Post by lobofta »

ileonichwiesz wrote: post #10

This might be an unkind reading, but to me this just sounds like an attempt to reinvent the very same kind of mysticism that it mentions in the first paragraph. “No need to study the world around you and wonder about its rules, peasant - it’s far beyond your understanding! Only ~the gods~ computers can ever know the truth!” I shudder to think about a future where people give up on working to understand complex system…

Might we ever distinguish what is complex and complicated? Probably not, but I guess the author argues that this gives us a way forward because we can try to distill large models.

Re: Billion-Parameter Theories

#12

Post by b450 »

Reminds me of the blog post about Waymo's "World Model". Training on real-world data results in a sufficiently rich model to start simulating novel scenarios that aren't in the training data (like the elephant wandering into the street), which in turn can feed back into training. One could imagine scientific inquiry working the same way.

It strikes me that many of these complex systems have indeterminate boundaries, and a fair amount of distortion might be baked into the choice of training data. Poverty (to take an example from this post) probably has causes at economic, psychological, ecological, physiological, historical, and political levels of description (commenters please note I didn't think too hard about this list). What data we feed into our models, and how those data are understood as operationalizations of the qualitative phenomena we care about, might matter.

Re: Billion-Parameter Theories

#14

Post by seanlinehan »

curuinor wrote: post #6

Connectionist models have lots of theory by theoreticians explicitly pissed off about Chomsky's assertion that there is an inbuilt ability for language. Jay McClelland's office had a little corkboard thingy with Chomsky mockery on the side, for example. Putting forth even the implicature that the present direct descendants are intellectual descendants of Chomsky is like saying Protestants are intellectual descendants…

Perhaps a failure of communication -- I was indeed attempting to say that Chomsky was wrong and his ideas were interesting, but more or less a dead end.

Re: Billion-Parameter Theories

#15

Post by seanlinehan »

dakiol wrote: post #8

> You could capture the behavior of every falling object on Earth in three variables and describe the relationship between matter and energy in five characters. What we can do is to approximate. Newton had a good approximation some time ago about gravitation (force equals a constant times two masses divided by distance squared. Super readable indeed) But nowadays there's a better one that doesn't look like Newton's t…

Agreed!

Re: Billion-Parameter Theories

#16

Post by seanlinehan »

ileonichwiesz wrote: post #10

This might be an unkind reading, but to me this just sounds like an attempt to reinvent the very same kind of mysticism that it mentions in the first paragraph. “No need to study the world around you and wonder about its rules, peasant - it’s far beyond your understanding! Only ~the gods~ computers can ever know the truth!” I shudder to think about a future where people give up on working to understand complex system…

Not the intention at all. The part about mechanistic interpretability was meant to gesture at how building such systems can provide new tool kit for building further intuition and understanding.

Re: Billion-Parameter Theories

#17

Post by gwerbin »

b450 wrote: post #12

Reminds me of the blog post about Waymo's "World Model". Training on real-world data results in a sufficiently rich model to start simulating novel scenarios that aren't in the training data (like the elephant wandering into the street), which in turn can feed back into training. One could imagine scientific inquiry working the same way. It strikes me that many of these complex systems have indeterminate boundaries,…

This "world model" concept has been a big deal in AI research, in LLMs.

Re: Billion-Parameter Theories

#18

Post by seanlinehan »

niemandhier wrote: post #7

He talks about the Santa Fe institute and how they failed to carry their findings into the real world. They did not. They showed that for certain problems one could not do more than figure out some invariant and scaling laws. Showing what is impossible is not failure. For the rest: Modern gene networks and lots of biological modelling is based on their work as well as quite a few other things. That’s also not failure…

True -- I didn't mean to communicate that Santa Fe was a failure writ large. Their contribution was very important!

Though I think it's fair to say that the torch was picked up and carried by others with a different set of strategies.

Re: Billion-Parameter Theories

#19

Post by b450 »

dakiol wrote: post #8

> You could capture the behavior of every falling object on Earth in three variables and describe the relationship between matter and energy in five characters. What we can do is to approximate. Newton had a good approximation some time ago about gravitation (force equals a constant times two masses divided by distance squared. Super readable indeed) But nowadays there's a better one that doesn't look like Newton's t…

This kind of view tends to logically conclude in the idea of a noumenal, unknowable reality. I think it's more reasonable to say that truth itself is gold star we award to descriptions that suit our purposes. After all, descriptions are necessarily approximations (or reductive or "compressions"), since the only model of a thing with 100% fidelity is... the thing itself.

Re: Billion-Parameter Theories

#20

Post by lkm0 »

It's an optimistic point of view. Still, when people use large neural nets to model physics, they also have a lot of parameters but they replicate very simple laws. So there's something deeper about this. Something like a simulation of theory.
Post reply on HN