What is the status on continual learning for LLMs?

The difference with something completely new at inference time, that was not in the training data, or not used by RL training, is that while it will be able to use it to some extent, it has not been taught via RL how to best use in a reasoning chain.

The problem is that LLMs don't really have the generic ability to reason, so they instead need to fake it by either:

1) Fine tuning on reasoning data (very specific)

2) RL training (more generalizable, especially for math/coding)

3) Prompting that encourages "keep on going" "tree of thoughts" exploration where they may discover a reasoning chain largely by luck

Demis Hassabis has talked about combining LLMs with search (cf systems like AlphaGo) which sounds like it might be useful for math research.

I don't know what it would take to have an AI that might make a brand new discovery/theory like this... part of it would be the mechanics of continual learning, curiosity, etc, but maybe also part education/mentorship?. How/when does a human mathematician, tackling a problem and getting nowhere (or maybe via some other motivation), end up inventing a whole new framework/paradigm to make progress ?!

I think it would be very surprising to see today's AI (LLMs) do this, because there is just too much missing. LLMs were designed only to be next token predictors - i.e. copying/automation machines. There are trained to predict, hence copy, what they have seen before, not to innovate (other than by combining what they have already seen). Humans may learn and process language in a way similar to an LLM, based on prediction, and our brains are certainly evolved for prediction, but there is a lot more to us than that. At least half of our cortex is dedicated to feedback and learning - what to do when our predictions are wrong. LLMs don't have this. When an LLM prediction "fails" it hallucinates. When a human prediction fails other mechanisms kick in (curiosity, boredom, frustration), some purely cognitive, some emotional, and we potentially explore, innovate and overcome.

Presumably if you try the RAG (vector store) approach, then one necessary component would be for the system to continually review what it has already learnt and consolidate/expand that by looking for generalizations, exceptions, contradictions, implications, etc. In fact maybe this would be the core of the system, with subagents assigned to each of these (and more) tasks. You would just "seed" the memory with the task you wanted it to work on, some hints on direction, difficulties, etc, then let it get to work. This would be rather like a GOFAI "blackboard" system where a bunch of loosely collaborating agents make progress on a problem by using a shared blackboard to get new work and post results (which then become inputs to other agents).