Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MemGPT – LLMs with self-editing memory for unbounded context

github.com

Post reply on HN
Print view View original

41–50 of 90 posts

Re: MemGPT – LLMs with self-editing memory for unbounded context

#41

Post by shishirpatil »

Difwif wrote: post #25

I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.

Yeah! While it’s not known what close-sourced models do, what we think is happening based on some prompt attacks, is that they also use recursive summarization (in addition to what others have mentioned in this thread).

Re: MemGPT – LLMs with self-editing memory for unbounded context

#42

Post by pacjam »

Difwif wrote: post #25

I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.

Recursive summarization is a simple and popular way to provide the illusion of infinite context (when you need to free up space, just summarize the oldest N messages into 1 summary message). It's lossy and you'll inevitably lose important information, but it should degrade relatively gracefully. In MemGPT we use (implicit) recursive summarization on top of all the explicit memory management.

Re: MemGPT – LLMs with self-editing memory for unbounded context

#43

Post by shishirpatil »

littlestymaar wrote: post #34
wilg wrote: post #30

I was just suggesting something like this to a friend yesterday! (Neither of us know enough to do it or know if it's a good idea.) However, I do think the context length is one of the top improvements that would make LLMs much more useful.

Same here, it looks like the idea was pretty obvious. Glad to see it implemented though. Context length being so limited is the number one thing that rules LLM as possessing something that resemble “intelligence”, so if we this kind of unbounded context length we're entering into a completely new universe in terms of LLM abilities.

Thanks @wilg and @littlestymaar ! Yeah totally, with the benefit of hindsight, this makes total sense! Hope you find some of our codebase useful to build on top of. We are an Apache 2.0 licensed open source project and welcome contributions :)

Re: MemGPT – LLMs with self-editing memory for unbounded context

#44

Post by pacjam »

Tostino wrote: post #39

Heh, I've been working on...a good portion of the basics that this project / paper have tested out for the past few months as an idea (as I work more on other more material problems for my side project). I have a whole document of my thoughts on this topic, and as I was reading through the paper just piece after piece of the concepts that I had documented kept coming up. Glad I am not the only one thinking in this di…

Happy to chat more about other ideas in this direction! There are plenty of things we tried with varying degrees of success (especially when trying to get MemGPT to work on less powerful LLMs), and we'd be interested in hearing what you observed in your own work.

Re: MemGPT – LLMs with self-editing memory for unbounded context

#46

Post by ASalazarMX »

pacjam wrote: post #42
Difwif wrote: post #25

I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.

Recursive summarization is a simple and popular way to provide the illusion of infinite context (when you need to free up space, just summarize the oldest N messages into 1 summary message). It's lossy and you'll inevitably lose important information, but it should degrade relatively gracefully. In MemGPT we use (implicit) recursive summarization on top of all the explicit memory management.

Would this be the same method used to assign a title to your chat based on the first prompt? It's surprisingly effective at getting the core idea most of the time.

Re: MemGPT – LLMs with self-editing memory for unbounded context

#47

Post by pacjam »

ASalazarMX wrote: post #46
pacjam wrote: post #42

Earlier quoted context omitted.

Recursive summarization is a simple and popular way to provide the illusion of infinite context (when you need to free up space, just summarize the oldest N messages into 1 summary message). It's lossy and you'll inevitably lose important information, but it should degrade relatively gracefully. In MemGPT we use (implicit) recursive summarization on top of all the explicit memory management.

Would this be the same method used to assign a title to your chat based on the first prompt? It's surprisingly effective at getting the core idea most of the time.

Thanks for your interest! Question - does the title of the chat ever change after it's first assigned? If so, using a recursive summary to refresh the title sounds like a reasonable idea (especially if you're already computing a summary to extend context).

From what I remember the title in ChatGPT gets set once after a few messages, in which case I'd assume it's generated with a special "title generation" prompt (that gets the first few messages as input).

In either case since I don't work at OpenAI I can't tell you for sure ;)

Re: MemGPT – LLMs with self-editing memory for unbounded context

#48

Post by visarga »

Der_Einzige wrote: post #28
Difwif wrote: post #25

I've had a suspicion for a while now that this is what ChatGPT does within a conversation (chat.openai.com, not the api). I've had very long chat histories that seem to gracefully degrade instead of just forgetting everything. Maybe there's more clues in the context than I realize though. Either way this type of idea will probably be a fundamental feature for all chat bots in the future IMO.

ChatGPT degrades precisely because they aren't doing anything special to extend their memory beyond the context length. There are trivial techniques to implement "lossy" memory, such as just average pooling tokens (the same approach used by sentence transformers). Not sure why it's so rare to see this used for condensing a huge amount of context into a prompt. It is effectively "medium" term memory.

At the very least I would average vectors inside single words or word compounds getting a 2-3x reduction in length without much work.
Post reply on HN