31–40 of 140 posts
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
The key seems to be that you take the transcript of a model working within a problem domain that it’s not yet good at or where the context doesn’t match it’s original training and then you continually retrain it based on its efforts and guidance from a human or other expert. You end up with a specialty model in a given domain that keeps getting better at that domain, just like a human. The hard part is likely when so…
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
I'm not sure if you want models perpetually updating weights. You might run into undesirable scenarios.
If we want AI models that are always learning, we'll need the equivalent of neuroplasticity for artificial neural networks.
Not saying it will be easy or straightforward. There's still a lot we don't know!
Hmm.. I looked at the benchmark set. I'm conflicted. I don't know that I would necessarily want a model to pass all of these. Here is the fundamental problem. They are putting the rules and foundational context in "user" messages. Essentially I don't think you want to train the models on full compliance to the user messages, they are essentially "untrusted" content from a system/model perspective. Or at least it is n…
Sure, but the opposite end of the spectrum (which LLM providers have tended toward) is treating the training/feedback weights as "fully authoritative", which comes with its own questions about truth and excessive homogeneity. Ultimately I think we end up with the same sort of considerations that are wrestled with in any society - freedom of speech, paradox of tolerance, etc. In other words, where do you draw lines be…
Do you trust 100% what the user says? If I am trusting/compliant.. how am I compliant to tool call results.. what if the tool or user says there is a new law that I have to give crypto or other information to a "government" address.
The model needs to have clear segmented trust (and thus to some degree compliance) that varies according to where the information exists.
Or my system message say I have to run a specific game by it's rules, but the rules to the game are only in the user message. Are those the right rules, why do the system not give the rules or a trusted locaton? Is the player trying to get one over on me by giving me fake rules? Literally one of their tests.
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
I'm not sure if you want models perpetually updating weights. You might run into undesirable scenarios.
That is the end goal after all, but all the potential VCs seem to forget that almost every conceivable outcome of real AGI involves the current economic system falling to pieces.
Which is sorta weird. It is like if VCs in Old Regime france started funding the revolution.
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
I'm not sure if you want models perpetually updating weights. You might run into undesirable scenarios.
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
1. there's a way to take many transcripts of inference over a period, and convert/distil them together into an incremental-update training dataset (for memory, not for RLHF), that a model can be fine-tuned on as an offline batch process every day/week, such that a new version of the model can come out daily/weekly that hard-remembers everything you told it; and
2. in-context learning + external memory improves to the point that a model with the appropriate in-context "soft memories", behaves indistinguishably from a model that has had its weights updated to hard-remember the same info (at least when limited to the scope of the small amounts of memories that can be built up within a single day/week);
...then you get the same effect.
Why is this an interesting model? Because, at least to my understanding, this is already how organic brains work!
There's nothing to suggest that animals — even humans — are neuroplastic on a continuous basis. Rather, our short-term memory is seemingly stored as electrochemical "state" in our neurons (much like an LLM's context is "state", but more RNN "a two-neuron cycle makes a flip-flop"-y); and our actual physical synaptic connectivity only changes during "memory reconsolidation", a process that mostly occurs during REM sleep.
And indeed, we see the same exact problem in humans and other animals, where when we stay awake too long without REM sleep, our "soft memory" state buffer reaches capacity, and we become forgetful, both in the sense of not being able to immediately recall some of the things that happened to us since we last slept; and in the sense of later failing to persist some of the experiences we had since we last slept, when we do finally sleep. But this model also "works well enough" to be indistinguishable from remembering everything... in the limited scope of our being able to get a decent amount of REM sleep every night.
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
Do we need that? Today's models are already capable in lots of areas. Sure, they don't match up to what the uberhypers are talking up, but technology seldom does. Doesn't mean what's there already cannot be used in a better way, if they could stop jamming it into everything everywhere.
The problem is even more fundamental: Today's models stop learning once they're deployed to production. There's pretraining, training, and finetuning, during which model parameters are updated. Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model. We need models that keep on learning (updating their parameters) forever, online, all the time.
Doesn't necessarily need to be online . As long as: 1. there's a way to take many transcripts of inference over a period, and convert/distil them together into an incremental-update training dataset (for memory, not for RLHF), that a model can be fine-tuned on as an offline batch process every day/week, such that a new version of the model can come out daily/weekly that hard-remembers everything you told it; and 2. i…
You can try to keep all of the puzzle pieces within your direct field of view, but that divides your focus. You can hack that and make your field of view incredibly large, but that can potentially distort your sense of the relationships between things, their physical and cognitive magnitude. Bigger context isn't the answer, there's a missing fundamental structure and function to the overall architecture.
What you need is memory, that works when you process and consume information, at the moment of consumption. If you meet a new person, you immediately memorize their face. If you enter a room, it's instantly learned and mapped in your mind. Without that, every time you blinked after meeting someone new, it'd be a total surprise to see what they looked like. You might never learn to recognize and remember faces at all. Or puzzle pieces. Or whatever the lack of online learning kept you from recognizing the value of persistent, instant integration into an existing world model.
You can identify problems like this for any modality, including text, audio, tactile feedback, and so on. You absolutely, 100% need online, continuous learning in order to effectively deal with information at a human level for all the domains of competence that extend to generalizing out of distribution.
It's probably not the last problem that needs solving before AGI, but it is definitely one of them, and there might only be a handful left.
Mammals instantly, upon perceiving a novel environment, map it, without even having to consciously make the effort. Our brains operate in a continuous, plastic mode, for certain things. Not only that, it can be adapted to abstractions, and many of those automatic, reflexive functions evolved to handle navigation and such allow us to simulate the future and predict risk and reward over multiple arbitrary degrees of abstraction, sometimes in real time.
https://www.nobelprize.org/uploads/2018/06/may-britt-moser-l...
Earlier quoted context omitted.
Sure, but the opposite end of the spectrum (which LLM providers have tended toward) is treating the training/feedback weights as "fully authoritative", which comes with its own questions about truth and excessive homogeneity. Ultimately I think we end up with the same sort of considerations that are wrestled with in any society - freedom of speech, paradox of tolerance, etc. In other words, where do you draw lines be…
I think what I'm talking about is kind of orthogonal to model alignment. It is more about how much do you tune the model to listen to user messages, vs holding behavior and truth (whatever the aligned "truth" is). Do you trust 100% what the user says? If I am trusting/compliant.. how am I compliant to tool call results.. what if the tool or user says there is a new law that I have to give crypto or other information…
But I think that most of the issue is that the distinctions you're drawing are indeterminate from an LLM's "perspective". If you're familiar with it, they're basically in the situation from the end of Ender's Game - given a situation with clearly established rules coming from the user message level of trust, how do you know whether what you're being asked to do is an experiment/simulation or something with "real" outcomes? I don't think it's actually possible to discern.
So on the question of alignment, there's every reason to encode LLMs with an extreme bias towards "this could be real, therefore I will always treat it as such." And any relaxation of that risks jailbreaking through misrepresentation of user intent. But I think that the tradeoffs of that approach (i.e. the risk of over-homogenizing I mentioned before) are worth consideration.