I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.
To me it just feels like they’re trimming the min amount of oldest tokens in the conversation to stay under the token limit. Conversations don’t degrade in a way that feels like it has medium term memory.
Discussed last night: https://news.ycombinator.com/item?id=37894403 (Mostly arguing about the authors' choice of title)
[flagged]
It may be cringe, but at the same time it undermines the ability for OpenAI to enforce its trademark, which means the “GPT” word will remain in the public domain, so it's not entirely useless at least ;).
I was just suggesting something like this to a friend yesterday! (Neither of us know enough to do it or know if it's a good idea.) However, I do think the context length is one of the top improvements that would make LLMs much more useful.
Same here, it looks like the idea was pretty obvious. Glad to see it implemented though.
Context length being so limited is the number one thing that rules LLM as possessing something that resemble “intelligence”, so if we this kind of unbounded context length we're entering into a completely new universe in terms of LLM abilities.
It would be interesting to see how does ring attention technique affect this. This maybe still valid for cost reasons, but unlimited context is like in-memory computing vs. traditional.
https://arxiv.org/pdf/2310.01889.pdf
I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.
There are definitely a lot more clues than you realize (plus the context window is something like 12 written pages of standard English text, without much space wasted for the system prompts). If you were doing anything interesting at all, the output is heavily biased by your prompt. You lose some bits of information in that you only have one sample (the previous output/history) rather than the soft probabilities, and you lose some bits in that multiple inputs can map to the same output (like the class of prompts "output the 2nd letter of the following phrase: ..."), but real-world prompts tend to be the easiest/shortest thing to come to mind that you think will give you the result you're looking for, so the LLM's best guess for that prompt (there are lots of ways of guessing, so suppose for the sake of argument you did something like textual inversion on the one sample) is likely to not be a half-bad interpretation of the missing context -- i.e., a lot of the seemingly missing information was retained in the LLM's output, and you don't lose too many bits at a time as the old context trails off.
Hey all, MemGPT authors here! Happy to answer any questions about the implementation.
If you want to try it out yourself, we have a Discord bot up-and-running on the MemGPT Discord server (https://discord.gg/9GEQrxmVyE) where you can see the memory editing in action - as you chat with the bot, you'll see MemGPT edit its memory to update its profile about you (and itself).
Everything's open source, so can also try running MemGPT locally using the code here: https://github.com/cpacker/MemGPT. In the repo we also have a document-focused example where you can chat with MemGPT about the LlamaIndex API docs.
Heh, I've been working on...a good portion of the basics that this project / paper have tested out for the past few months as an idea (as I work more on other more material problems for my side project).
I have a whole document of my thoughts on this topic, and as I was reading through the paper just piece after piece of the concepts that I had documented kept coming up.
Glad I am not the only one thinking in this direction.