Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning from context is harder than we thought

hy.tencent.com

Post reply on HN
Print view View original

61–70 of 140 posts

Re: Learning from context is harder than we thought

#61

Post by bluegatty »

observationist wrote: post #39
derefr wrote: post #37

Earlier quoted context omitted.

Doesn't necessarily need to be online . As long as: 1. there's a way to take many transcripts of inference over a period, and convert/distil them together into an incremental-update training dataset (for memory, not for RLHF), that a model can be fine-tuned on as an offline batch process every day/week, such that a new version of the model can come out daily/weekly that hard-remembers everything you told it; and 2. i…

It 100% needs to be online. Imagine you're trying to think about a new tabletop puzzle, and every time a puzzle piece leaves your direct field of view, you no longer know about that puzzle piece. You can try to keep all of the puzzle pieces within your direct field of view, but that divides your focus. You can hack that and make your field of view incredibly large, but that can potentially distort your sense of the r…

That's not how training works - adjusting model weights to memorize a single data item is not going to fly.

Model weights store abilities, not facts - generally.

Unless the fact is very widely used and widely known, with a ton of context around it.

The model can learn the day JFK died because there are millions of sparse examples of how that information exists in the world, but when you're working on a problem, you might have 1 concern to 'memorize'.

That's going to be something different than adjusting model weights as we understand them today.

LLMs are not mammals either, it's helpful analogy in terms of 'what a human might find useful' but not necessary in the context of actual llm architecture.

The fact is - we don't have memory sorted out architecturally - it's either 'context or weights' and that's that.

Also critically: Humans do not remember the details of the face. Not remotely. They're able to associate it with a person and name 'if they see it again' - but that's different than some kind of excellent recall. Ask them to describe features in detail and maybe we can't do it.

You can see in this instance, this may be related to kind of 'soft lookup' aka associating an input with other bits of information which 'rise to the fore' as possibly useful.

But overall, yes, it's fair to take the position that we'll have to 'learn from context in some way'.

Re: Learning from context is harder than we thought

#63

Post by observationist »

bluegatty wrote: post #61
observationist wrote: post #39

Earlier quoted context omitted.

It 100% needs to be online. Imagine you're trying to think about a new tabletop puzzle, and every time a puzzle piece leaves your direct field of view, you no longer know about that puzzle piece. You can try to keep all of the puzzle pieces within your direct field of view, but that divides your focus. You can hack that and make your field of view incredibly large, but that can potentially distort your sense of the r…

That's not how training works - adjusting model weights to memorize a single data item is not going to fly. Model weights store abilities, not facts - generally. Unless the fact is very widely used and widely known, with a ton of context around it. The model can learn the day JFK died because there are millions of sparse examples of how that information exists in the world, but when you're working on a problem, you m…

I suspect we're going to need hypernetworks of some sort - dynamically generated weights, with the hypernet weights getting the dream-like reconsolidation and mapping into the model at large, and layers or entire experts generated from the hypernets on the fly, a degree removed from the direct-from-weights inference being done now. I've been following some of the token-free latent reasoning and other discussions around CoT, other reasoning scaffolding, and so forth, and you just can't overcome the missing puzzle piece problem elegantly unless you have online memory. In the context of millions of concurrent users, that also becomes a nightmare. Having a pipeline, with a sort of intermediate memory, constructive and dynamic to allow resolution of problems requiring integration into memorized concepts and functions, but held out for curation and stability.

It's an absolutely enormous problem, and I'm excited that it seems to be one of the primary research efforts kicking off this year. It could be a very huge capabilities step change.

Re: Learning from context is harder than we thought

#64

Post by observationist »

bluegatty wrote: post #61
observationist wrote: post #39

Earlier quoted context omitted.

It 100% needs to be online. Imagine you're trying to think about a new tabletop puzzle, and every time a puzzle piece leaves your direct field of view, you no longer know about that puzzle piece. You can try to keep all of the puzzle pieces within your direct field of view, but that divides your focus. You can hack that and make your field of view incredibly large, but that can potentially distort your sense of the r…

That's not how training works - adjusting model weights to memorize a single data item is not going to fly. Model weights store abilities, not facts - generally. Unless the fact is very widely used and widely known, with a ton of context around it. The model can learn the day JFK died because there are millions of sparse examples of how that information exists in the world, but when you're working on a problem, you m…

Also, with regards to faces, that's kind of what I'm getting at - we don't have grid cells for faces, there seem to be discrete, functional, evolutionary structures and capabilities that combine in ways we're not consciously aware of to provide abilities. We're reflexively able to memorize faces, but to bring that to consciousness isn't automatic. There've been amnesia and lesion and other injury studies where people with face blindness get stress or anxiety, or relief, when recognizing a face, but they aren't consciously aware. A doctor, or person they didn't like, showing up caused stress spikes, but they couldn't tell you who they were or their name, and the same with family members- they get a physiological, hormonal response as if they recognized a friend or foe, but it never rises to the level of conscious recognition.

There do seem to be complex cells that allow association with a recognizable face, person, icon, object, or distinctive thing. Face cells apply equally to abstractions like logos or UI elements in an app as they do to people, famous animals, unique audio stings, etc. Split brain patients also demonstrate amazing strangeness with memory and subconscious responses.

There are all sorts of layers to human memory, beyond just short term, long term, REM, memory palaces, and so forth, and so there's no simple singular function of "memory" in biological brains, but a suite of different strategies and a pipeline that roughly slots into the fuzzy bucket words we use for them today.

Re: Learning from context is harder than we thought

#65

Post by nimonian »

em-bee wrote: post #20
numtel wrote: post #16

Earlier quoted context omitted.

There's a Terence McKenna quote about this: > So, for instance, you know, I’ve made this example before: a child lying in a crib and a hummingbird comes into the room and the child is ecstatic because this shimmering iridescence of movement and sound and attention, it’s just wonderful. I mean, it is an instantaneous miracle when placed against the background of the dull wallpaper of the nursery and so forth. But, the…

and what is this quote supposed to explain? that language prevents a child from learning nuance? sounds like nonsense to me. a child first learns broad categories. for example some children as they learn to speak think every male person is dad. then they recognize everyone with a beard is dad, because dad has a beard. and only later they learn to differentiate that dad is only one particular person. same goes for the…

Wittgenstein famously said "The limits of my language mean the limits of my world."

Alan Watts suggests people like Wittgenstein should occasionally try to let go of this way of thinking. Apologies if it is sentimental but I hope you'll give him a chance, it's quite short: https://m.youtube.com/watch?v=heksROdDgEk

In reflection of all of this, I think that the quote you're responding to only meant to say that experiencing the world through language means building an abstraction over its richness. (I somewhat agree with you, though, that the quote seems a little dramatic. Maybe that's just my taste.)

One more thought.

I think there's a reason why various forms of meditation teach us to stop thinking. Maybe they are telling us to sometimes stop dealing with our abstractions, powerful though they might be, and experience the real thing once in a while.

Re: Learning from context is harder than we thought

#66

Post by bluegatty »

observationist wrote: post #63
bluegatty wrote: post #61

Earlier quoted context omitted.

That's not how training works - adjusting model weights to memorize a single data item is not going to fly. Model weights store abilities, not facts - generally. Unless the fact is very widely used and widely known, with a ton of context around it. The model can learn the day JFK died because there are millions of sparse examples of how that information exists in the world, but when you're working on a problem, you m…

I suspect we're going to need hypernetworks of some sort - dynamically generated weights, with the hypernet weights getting the dream-like reconsolidation and mapping into the model at large, and layers or entire experts generated from the hypernets on the fly, a degree removed from the direct-from-weights inference being done now. I've been following some of the token-free latent reasoning and other discussions arou…

Yes, so I think that's a fine thought, I don't think it fits into LLM architecture.

Also, weirdly, even Lecun etc. are barely talking about this, they're thinking about 'world models etc'.

I think what you're talking about is maybe 'the most important thing' right now, and frankly, it's almost like an issue of 'Engineering'.

Like - its when you work very intently with the models so this 'issue' become much more prominent.

Your 'instinct' for this problem is probably an expression of 'very nuanced use' I'm going to guess!

So in a way, it's as much Engineering as it is theoretical?

Anyhow - so yes - but - probably not LLM weights. Probably.

I'll add a small thing: the way that Claude Code keeps the LLM 'on track' is by reminding it! Literally, it injects little 'TODO reminders' with some prompts, which is kind of ... simple!

I worked a bit with 'steering probes' ... and there's a related opportunity there - to 'inject' memory and control operations along those lines. Just as a starting point for a least one architectural motivation.

Re: Learning from context is harder than we thought

#67

Post by furyofantares »

cs702 wrote: post #24

The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.

Why is learning an appropriate metaphor for changing weights but not for context? There are certainly major differences in what they are good or bad at and especially how much data you can feed them this way effectively. They both have plenty of properties we wish the other had. But they are both ways to take an artifact that behaves as if it doesn't know something and produce an artifact that behaves as if it does.

I've learned how to solve a Rubik's cube before, and forgot almost immediately.

I'm not personally fond of metaphors to human intelligence now that we are getting a better understanding of the specific strengths and weaknesses these models have. But if we're gonna use metaphors I don't see how context isn't a type of learning.

Re: Learning from context is harder than we thought

#68

Post by rustyhancock »

It's a very interest benchmark. Much more impressive than needle in haystack benches or just tuneable benches.

I wonder if it's somewhat incompatible with some domains.

I.e. perhaps coding models need to rigidly stick to what they know and resist bad ideas in their contexts - I don't want my mistakes to be replicated by the model.

Still I agree with the premise that learning in session is what I want from a model.

Perhaps once models mature they will diverge even more than just having sophistication and coding or not. But creative, coding, rule based etc models

Re: Learning from context is harder than we thought

#69

Post by BobbyTables2 »

cs702 wrote: post #24

The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.

How long will it take someone to poison such a model by teaching it wrong things?

Even humans fall for propaganda repeated over and over .

The current non-learning model is unintentionally right up there with the “immutable system” and “infrastructure as code” philosophy.

Re: Learning from context is harder than we thought

#70

Post by BobbyTables2 »

com2kid wrote: post #46
CorrectHorseBat wrote: post #41

Earlier quoted context omitted.

Yes the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders. And for your comparison, they did fund the American revolution which on its turn was one of the sparks for the French revolution (or was that exactly the point you were making?)

The funding of the American revolution is a fun topic but most people don't know about it so I don't bother dropping references to it. :D

I wonder which side tried to forget that first (;->
Post reply on HN