51–60 of 90 posts
the link and title of this suspiciously changed. @dang
I was just suggesting something like this to a friend yesterday! (Neither of us know enough to do it or know if it's a good idea.) However, I do think the context length is one of the top improvements that would make LLMs much more useful.
The high-level memory read/writes are quite intuitive and you may be surprised at how closely it matches what you were suggesting to your friend.
Heh, I've been working on...a good portion of the basics that this project / paper have tested out for the past few months as an idea (as I work more on other more material problems for my side project). I have a whole document of my thoughts on this topic, and as I was reading through the paper just piece after piece of the concepts that I had documented kept coming up. Glad I am not the only one thinking in this di…
Happy to chat more about other ideas in this direction! There are plenty of things we tried with varying degrees of success (especially when trying to get MemGPT to work on less powerful LLMs), and we'd be interested in hearing what you observed in your own work.
The title made me think this was an approach that used memory editing techniques (e.g. ROME [1]) to allow an LLM's neural memory (not just its context) to change over the course of conversation. Pretty happy to realize that this is just a fancy RAG work—will be building my version of MemEditGPT soon. [1] https://arxiv.org/abs/2202.05262
You're correct that MemGPT doesn't do editing of LLM weights like in ROME - the "memory" we're considering in MemGPT is at the text/token level, not the weight level. The core concepts behind MemGPT is giving the LLM the ability to edit a working memory scratchpad (held in-context) and reading/writing to external context via functions. An important detail is that reads are always paginated (chunked) to deal with finite context limits, and MemGPT can do many iterative read/writes from a single user input (by chaining functions together). This allows MemGPT to search over a large database of documents for example, collecting information from various sources to return an answer (as in our LlamaIndex API docs example on the README).
I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.
Recursive summarization is a simple and popular way to provide the illusion of infinite context (when you need to free up space, just summarize the oldest N messages into 1 summary message). It's lossy and you'll inevitably lose important information, but it should degrade relatively gracefully. In MemGPT we use (implicit) recursive summarization on top of all the explicit memory management.
I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.
I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.
ChatGPT degrades precisely because they aren't doing anything special to extend their memory beyond the context length. There are trivial techniques to implement "lossy" memory, such as just average pooling tokens (the same approach used by sentence transformers). Not sure why it's so rare to see this used for condensing a huge amount of context into a prompt. It is effectively "medium" term memory.
Fed chatGPT special numbers, then 3k tokens, then 2k tokens. after that, it was unable to understand any question about the special numbers provided.