A Bitter Lesson for Memory
personal-website-3bed.onrender.com
A Bitter Lesson for Memory
1–4 of 4 posts
Re: A Bitter Lesson for Memory
#2If understanding emerges from pre-training, then perhaps memory is what emerges from post-training.
Re: A Bitter Lesson for Memory
#3I've always been amazed at how terrible most frontier LLMs are at compaction given how embarrassingly easy it is to come up with half a dozen different RL training evals which would teach models to generate useful context summaries. Heck, you could bolt it onto any existing RL eval by just forcing a compaction every three turns.
Re: A Bitter Lesson for Memory
#4I've always been amazed at how terrible most frontier LLMs are at compaction given how embarrassingly easy it is to come up with half a dozen different RL training evals which would teach models to generate useful context summaries. Heck, you could bolt it onto any existing RL eval by just forcing a compaction every three turns.
Yep. Or even better, compact after a random number of turns. The model must then learn to preserve useful context at arbitrary context lengths.