Natural language instructions induce generalization in networks of neurons
1–10 of 95 posts
Re: Natural language instructions induce generalization in networks of neurons
#2It's as if language is itself the latent space for these psychophysical tasks, especially compositional instruction. Their description of it as a scaffolding also seems apt.
Re: Natural language instructions induce generalization in networks of neurons
#3It's as if language is itself the latent space for these psychophysical tasks, especially compositional instruction. Their description of it as a scaffolding also seems apt.
I hate the reductive nature of the concept of "latent spaces".
A good enough formula for a task isn't a solution for every task. Yes Newtonian mechanics work, but Einstein is a better reflection of reality.
Re: Natural language instructions induce generalization in networks of neurons
#4> We found that language scaffolds sensorimotor representations such that activity for interrelated tasks shares a common geometry with the semantic representations of instructions, allowing language to cue the proper composition of practiced skills in unseen settings.
Sapir-Whorf with the surprise comeback?
Re: Natural language instructions induce generalization in networks of neurons
#5It's as if language is itself the latent space for these psychophysical tasks, especially compositional instruction. Their description of it as a scaffolding also seems apt.
I hate the reductive nature of the concept of "latent spaces". A good enough formula for a task isn't a solution for every task. Yes Newtonian mechanics work, but Einstein is a better reflection of reality.
I'm not sure I understand the analogy. The very idea of NNs is that it's not perfect, it is messy and not optimal, but is very generalizable.
Re: Natural language instructions induce generalization in networks of neurons
#6>Tasks that are instructed using conditional clauses also require a simple form of deductive reasoning (if p then q else s)
> Our models ofer several experimentally testable predictions outlining how linguistic information must be represented to facilitate flexible and general cognition in the human brain.
Aren't those claims falsified by more recent studies that show that even in flys, preferred direction to a moving stimulus uses the timing of spikes. And that fear conditioning in even mice uses Dendritic Compartmentalization?
Or that humans can even do xor with a single neuron.
If "must be represented" was "may be modeled by" I would have less of an issue and obviously spikey artificial NNs have had problems with riddled basins and make autograd problematic in general.
So ANNs need to be binary and it is best to model biological neurons as such for practical models... but can someone please clarify why 'must' can apply when using what we know now is an oversimplified artificial neuron models?
Here are a couple of recent papers but I think dendritic compartmentalization and spike timing sensitivity has been established for over a decade.
https://pubmed.ncbi.nlm.nih.gov/35701166/
https://www.sciencedirect.com/science/article/pii/S009286741...
Re: Natural language instructions induce generalization in networks of neurons
#7It's as if language is itself the latent space for these psychophysical tasks, especially compositional instruction. Their description of it as a scaffolding also seems apt.
I'm not pretending to understand half the words uttered in this discussion but I'm constantly reminded of how much it helps me to articulate things (explain them to others, write them down, etc) to understand them. Maybe that thinking indeed happens almost entirely on a linguistic level and I'm not doing half as much other thinking (visualization, abstract logic, etc.) in the process as I thought. That feels weird.
Re: Natural language instructions induce generalization in networks of neurons
#8TL;DR: The authors embed task instructions in a vector space with a language model, and train a sensorimotor-controlling model on top to perform tasks given the instruction embeddings. The authors find that the models generalize to previously unseen tasks, specified in natural language. Moreover, the authors show that the hidden states learn to represent task subcomponents, which helps explains why the model is able to generalize.
Re: Natural language instructions induce generalization in networks of neurons
#9>Tasks that are instructed using conditional clauses also require a simple form of deductive reasoning (if p then q else s) > Our models ofer several experimentally testable predictions outlining how linguistic information must be represented to facilitate flexible and general cognition in the human brain. Aren't those claims falsified by more recent studies that show that even in flys, preferred direction to a movin…
> Or that humans can even do xor with a single neuron.
That's news to me.
I'm not hugely surprised given I've heard a biological neuron is supposed to be equivalent to a small ANN network, but still, first I've heard of that claim.
Re: Natural language instructions induce generalization in networks of neurons
#10[deleted]