Natural language instructions induce generalization in networks of neurons • Hacker News Board

Re: Natural language instructions induce generalization in networks of neurons

#91

Post by mrblah » Wed, Mar 20, 2024, 2:55 PM UTC

tsimionescu wrote: post #84

mrblah wrote: post #36

Earlier quoted context omitted.

i've always assumed that language was required to give your brain the abstractions needed to reference things in the past compared to your current perception (aka now), like an index. if you think about your earliest memories, they almost certainly came after language. i'd be interested to know if any of the documented 'wild child' cases (infants 'raised by wolves') ever delved into what the children remembered befor…

There were efforts to teach them language as adolescents, but they didn't acquire it - as far as we know, it's not possible to acquire language if you don't do it as an infant. This is similar to other brain functions that aren't present at birth and require stimulation, such as sight. That is, if your eyes are forced closed for the first few months of your life, you will never be able to see, even if later they are…

i just had the uncomfortable thought that it's possible a disease could kill off everyone older than 1, sterilizing the species to language at some scale. for the few that survive, their world would be feral.

Re: Natural language instructions induce generalization in networks of neurons

#92

Post by nerdponx » Wed, Mar 20, 2024, 3:05 PM UTC

hackerlight wrote: post #79

nerdponx wrote: post #50

Earlier quoted context omitted.

The entire innovation (discovery?) of LLMs is that a good formula for the task of sequence completion turns out to also be a good formula for a wide range of AI tasks. That emergent property is why language models are called language models.

That's an inductive bias, not an emergent property. The inductive bias of transformers is that they're good at integrating global context from different parts of a sequence without a particular bias towards recent time steps or localized regularities. And that happens to be a good fit for many (but not all) real-world sequence learning tasks. The "emergent property" aspect is when LLMs are good at a task at scale X*3…

My point was that this particular inductive bias doesn't inherently beget a "language model".

Re: Natural language instructions induce generalization in networks of neurons

#93

Post by hackerlight » Wed, Mar 20, 2024, 3:27 PM UTC

nerdponx wrote: post #92

hackerlight wrote: post #79

Earlier quoted context omitted.

That's an inductive bias, not an emergent property. The inductive bias of transformers is that they're good at integrating global context from different parts of a sequence without a particular bias towards recent time steps or localized regularities. And that happens to be a good fit for many (but not all) real-world sequence learning tasks. The "emergent property" aspect is when LLMs are good at a task at scale X*3…

My point was that this particular inductive bias doesn't inherently beget a "language model".

It does beget a Transformer which we choose to call a language model when it's applied to language data

Re: Natural language instructions induce generalization in networks of neurons

#94

Post by mrblah » Wed, Mar 20, 2024, 6:13 PM UTC

tsimionescu wrote: post #84

mrblah wrote: post #36

Earlier quoted context omitted.

i've always assumed that language was required to give your brain the abstractions needed to reference things in the past compared to your current perception (aka now), like an index. if you think about your earliest memories, they almost certainly came after language. i'd be interested to know if any of the documented 'wild child' cases (infants 'raised by wolves') ever delved into what the children remembered befor…

There were efforts to teach them language as adolescents, but they didn't acquire it - as far as we know, it's not possible to acquire language if you don't do it as an infant. This is similar to other brain functions that aren't present at birth and require stimulation, such as sight. That is, if your eyes are forced closed for the first few months of your life, you will never be able to see, even if later they are…

Went down the rabbit hole and found the case of Danish Bear Boy, he was reportedly taught to speak but he claimed to have no memory of his time living with the bears. Fascinating stuff https://books.google.com/books?id=k2MRHJuQiVEC&dq=hesse+wolf...

Re: Natural language instructions induce generalization in networks of neurons

#95

Post by Teever » Thu, Mar 21, 2024, 8:56 PM UTC

Workaccount2 wrote: post #39

smokel wrote: post #20

Earlier quoted context omitted.

Hm, no. To start, cells are able to replicate themselves, whereas most silicon used today is not even close to doing so. The story you suggest seems to be built on a limited understanding of the processes involved. It's pretty hard to predict the future, especially given incorrect assumptions.

Compute is substrate independent. We use transistors because they are wicked fast and efficient. But a 4090 built from metal balls and wood blocks would still be able to perform all the same calculations. Or a 4090 made by drawing X's and O's on a (really massive) piece of paper. Or one made by connecting a bunch of neurons together for that matter. Saying cells can multiply doesn't really mean anything, unless is gi…

Compute may be substrate independent, but that isn't the point that the person you're replying to is refuting.

> It's clear that both biological sentient beings and sentient being made in factories in the future will essentially be two sides of the same coin...

> why should we consider the sentient being made of cells inherently superior to its transistor-based counterpart?

The defining characteristic of life as we know it, and unique quality of Earth as compared to other planets as we have measured them is the ability to self replicate, and the presence of self replicating matter.

Supposedly a single cell created all life on earth and is therefore directly responsible for every facet of the nature of Earth today. Over hundreds of millions of years the entire atmosphere, and entire surface of the earth as well as unmeasured parts of the sub surface of earth have been completely reshaped and defined by living cells all descended from that original cell.

We're very likely able to create human level intelligence or even super human level intelligence that operates in a factory produced robot body that can survive a human level life span with some sort of maintenance comparable to that of a mechanic/surgeon but if that being needs a factory to generate parts or progeny then it is intrinsically lacking compared to self-replicating cellular life.

Self-replication is a powerful ability that is intrinsic to life and deeply related to intelligent systems. As fabulous as a 4090 may be to us at this point in time, it is incomparable to a group of entities such as a humans that can create a civilization that can make a 4090, whatever material that 4090 may be made of.

A far more profoundly fascinating object is the factory that can make robots that can operate the factory that can make more of the robots, and better. And that's what multi-cellular life really is.

Until a 4090 type device can make more of itself and other by-products like multi-cellular life, the multi-cellular life that can make more of itself and by-products like a 4090 type device is intrinsically better.