Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pretraining Language Models via Neural Cellular Automata

hanseungwook.github.io

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Pretraining Language Models via Neural Cellular Automata

#11

Post by troelsSteegin »

voxleone wrote: post #2

Neural cellular automata are interesting because they shift learning from “predict tokens” to “model state evolution.” That feels much closer to a transition-based view of systems, where structure emerges from repeated local updates (transitions) rather than being encoded explicitly. I'm working on a theoretical/computational framework, the Functional Universe, intended for modeling physical reality as functional sta…

from https://voxleone.github.io/FunctionalUniverse/pages/executiv..., "The Functional Universe models reality as a history built from irreversible transitions, with time emerging from the accumulation of causal commitments rather than flowing as a primitive parameter." Is it fair to say that time is simply a way of organizing a log file on a dynamic reality? I interpreted "composition of transitions" as a system of processes. I think the hard modeling problem is interpreting interactions between processes - that transitions don't simply compose, that observed transitions may be confounded views of more complex transitions. I gather NCA would be granular enough to overcome that.

Re: Pretraining Language Models via Neural Cellular Automata

#12

Post by andai »

stanfordkid wrote: post #9

I did a similar project but using 3D fractals I found on shadertoy feeding into ViTs. They are extremely simple iterative functions that produce a ton of scene like complexity. I have a pet theory that the visual cortex when developing is linked to some kind of mechanism such as this. You just need proteins that create some sort of resonating signal that feed into the neurons as they grow (obviously this is hand-wavy…

I like your funny words, magic man!

Re: Pretraining Language Models via Neural Cellular Automata

#13

Post by andai »

> The key: since every sequence has a unique latent rule, the model must infer that rule in-context to predict what comes next. This in-context learning ability underpins many of the key reasoning capabilities observed in language models.

This is a remarkable paper. This is the first time I've heard someone training the actual thing we're trying to get this stuff to do!

---

> This raises a radical question: Is natural language the only path to intelligence?

Of course not! We have octopi, ravens etc., which in many domain display higher intelligence than frontier AIs.

"Embodied reasoning" (genetic algorithm brute force solving physical tasks for a billion years, to name one solution) is definitely one very practical form of intelligence, although we're taking some shortcuts in replicating it.

I'm wondering if simplified analog tasks like Box2D puzzled would help too (or perhaps even simpler? Hanoi? Block worlds?). I know many companies are using simulations of 3D worlds for that.

What I don't understand is how that can integrate with the LLM (physical intelligence would seem to require specialized circuitry, if only for the latency). But maybe once we have good specialized models, LLMs can be trained on their synthetic data?

Re: Pretraining Language Models via Neural Cellular Automata

#15

Post by heyitsguay »

stanfordkid wrote: post #9

I did a similar project but using 3D fractals I found on shadertoy feeding into ViTs. They are extremely simple iterative functions that produce a ton of scene like complexity. I have a pet theory that the visual cortex when developing is linked to some kind of mechanism such as this. You just need proteins that create some sort of resonating signal that feed into the neurons as they grow (obviously this is hand-wavy…

What were the results of 3d fractal shader pretraining?

Re: Pretraining Language Models via Neural Cellular Automata

#17

Post by gavinray »

Can someone ELI5 how this hypothesis could ever be true?

  > "The core hypothesis: what makes language useful for pre-training is its structure, not its semantics."
As a layman, I've always held the intuition that semantics are the only meaningful thing.

"Structure without semantics" = form without function, symmetric/regular noise, right?

My naive bet is on compressing semantics into mediums more expressive/information dense than text. Like how some languages have single words/symbols to represent entire sentence-long concepts.

Re: Pretraining Language Models via Neural Cellular Automata

#18

Post by voxleone »

troelsSteegin wrote: post #11
voxleone wrote: post #2

Neural cellular automata are interesting because they shift learning from “predict tokens” to “model state evolution.” That feels much closer to a transition-based view of systems, where structure emerges from repeated local updates (transitions) rather than being encoded explicitly. I'm working on a theoretical/computational framework, the Functional Universe, intended for modeling physical reality as functional sta…

from https://voxleone.github.io/FunctionalUniverse/pages/executiv... , "The Functional Universe models reality as a history built from irreversible transitions, with time emerging from the accumulation of causal commitments rather than flowing as a primitive parameter." Is it fair to say that time is simply a way of organizing a log file on a dynamic reality? I interpreted "composition of transitions" as a system of…

That’s a very good objection, and it’s pointing at a real pressure point in our framework.

Short answer: it’s close, but incomplete. It’s not that time organizes a log of reality; rather, reality is the accumulation of committed transitions. What you’re calling a ‘log’ it’s the ontological structure itself.

I gather you're basically saying: what we see as a transition ≠ what’s actually happening at the fundamental level. This is a legitimate and deep problem.

You’re right that observed transitions may not compose cleanly. In the Functional Universe, composition is a property of fundamental transitions. What we observe are often coarse-grained projections of many underlying transitions, which can obscure compositional structure.

Re: Pretraining Language Models via Neural Cellular Automata

#19

Post by andy12_ »

gavinray wrote: post #17

Can someone ELI5 how this hypothesis could ever be true? > "The core hypothesis: what makes language useful for pre-training is its structure, not its semantics." As a layman, I've always held the intuition that semantics are the only meaningful thing. "Structure without semantics" = form without function, symmetric/regular noise, right? My naive bet is on compressing semantics into mediums more expressive/informatio…

I think what they mean by this is that, for example, in "If it's raining the outside is wet. It's raining, so the outside is wet", it's more important for the model to learn "If A then B. A, therefore B" than to learn what "raining" , "outside" and "wet" mean.

Re: Pretraining Language Models via Neural Cellular Automata

#20

Post by al-yak »

Eternal question about pre-training, i.e. on what medium this pre-training or pre-pre-training has been performed ... if we assume that the platform is CMOS-based piece of hardware, then the most primitive trainable element would be a two-state learning automaton, that can be assembled out of a couple of dozen transistors. This is effectively a kind of a single cell bacteria. You then need to organize these automata into collectives which can start to exchange tokens and reinforcements between themselves and the training environment, and these collectives do so at a higher level and so on, and that's what can be seen as NCAs. (*) A good source of idea on this approach can be found in the book of M.L. Tsetlin: Automata Theory and Modelling of Biological Systems, 1973 - https://shop.elsevier.com/books/automation-theory-and-modeli...
Post reply on HN