I like to start with memory engineering then reach full system then reducing costs. This allows unlocking full potential of agents.
Interesting. Where can I read more about this?
Agentic Context Management: Memory and Cost as Architecture Problems
11–20 of 42 posts
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#12ACM, that's the term that I'd been looking for - and your paper explains it clearly. At the end, most of LLM problems are context problems. Getting the correct knowledge into its context window without overpopulating it is the actual engineering effort for most agents. And the solution you present seems promising. Both compaction with validation and predictive fetching are the way to go. I do not want to write an imp…
Thanks Samyakk!
1. Yes, works on docs, agent conversations, human-conversations from different sources (Slack, JIRA, etc.). We have connectors for some of these as well; so it is plug and play 2. conventional RAG recall accuracy is quite low (50-60%) and latency is pretty high (seconds). But worst is the precision; you end up context stuffing to get acceptable recall 3. We do offer on-prem deployments, but only on sizeable annual contracts
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#13Ive never read a paper cover to cover before but after wrestling with opus 5s english this paper is such a relief to read, its like my eyes has been washed off opus stink
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#14Context pollution and rot are probably more important than memory, because facts can usually be retrieved if the agent is good at following breadcrumbs. What's also the biggest killer is code rot. Agents are particularly good at death by thousand cuts. They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to…
Truly. Doing this for coding agents is an interesting and different shaped problem.
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#15Ive never read a paper cover to cover before but after wrestling with opus 5s english this paper is such a relief to read, its like my eyes has been washed off opus stink
Haha! I am going to put this one up as a win!
Thanks for reading! Hope you found it useful.
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#16ACM, that's the term that I'd been looking for - and your paper explains it clearly. At the end, most of LLM problems are context problems. Getting the correct knowledge into its context window without overpopulating it is the actual engineering effort for most agents. And the solution you present seems promising. Both compaction with validation and predictive fetching are the way to go. I do not want to write an imp…
[dead]
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#17Ive never read a paper cover to cover before but after wrestling with opus 5s english this paper is such a relief to read, its like my eyes has been washed off opus stink
Yet, it is full of AI slop one-liners like:
"The contest ahead is not over who stores the most data; it is over who manages context the best".
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#18[flagged]
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#19Context pollution and rot are probably more important than memory, because facts can usually be retrieved if the agent is good at following breadcrumbs. What's also the biggest killer is code rot. Agents are particularly good at death by thousand cuts. They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to…
> They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to copy from it on subsequent work. It spreads like a virus.
> Keeping these seeds out of the project is very difficult, and cleaning up the rot is very difficult.
My "aha" moment was when I realized this goes for all spheres of life where this tech is/will be introduced.
Re: Agentic Context Management: Memory and Cost as Architecture Problems
#20Context pollution and rot are probably more important than memory, because facts can usually be retrieved if the agent is good at following breadcrumbs. What's also the biggest killer is code rot. Agents are particularly good at death by thousand cuts. They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to…
> Then they continue to amplify that badness over time
Also, with "self-bias", models are also likely to grow new content into spots that match their subtle fingerprints from the past.
That might come at the expense of whatever corrected "we should avoid that and do this instead" alternative some human added for future architecture.