[dead]
I accidentally turned LLM memory into program analysis
11–20 of 88 posts
Re: I accidentally turned LLM memory into program analysis
#12Very cool article. I had a similar idea where "fact checking" should be real programs for logic correctness.
But IRL it's too vague. The exploit hunting is a better use case.
Re: I accidentally turned LLM memory into program analysis
#13Ctrl-F “prolog”: 0 result. :-/
Search on datalog instead.
Re: I accidentally turned LLM memory into program analysis
#14I reached a similar conclusion: LLMs should only really sit at the terminals of request fulfilment.
1. User request understanding: natural language -> a more rigorous representation, in my case Datalog.
2. Result interpretation: facts and derived facts -> natural language.
Between those terminals, the work should be mechanical reasoning over some ontology or formal knowledge structure.
That connects to another principle I've been thinking about, which I call Weathering: useful reasoning should change the shape of the system. If an LLM has already had to infer a relation, mapping, rule, or abstraction, repeated use should wear that inference into the system so that the next similar request doesn't require discovering it again from scratch.
With continued use, a weathering-capable system should therefore require less and less probabilistic intelligence for recurring work. Put another way, there should be a declining marginal cost of cognition since the products of intelligence harden into structure that can subsequently be reused and evaluated mechanically.
Re: I accidentally turned LLM memory into program analysis
#15This really matches up to my experience on long research projects with Claude.
It’s very hard to remove information - Claude has a habit of recording things all over the place and will happily treat things as facts even after they’ve been disproved.
What is currently true can get easily contaminated with old “facts”.
Re: I accidentally turned LLM memory into program analysis
#16It seems like you might be inventing a form of non-monotonic logic. Check out answer set programming, it actually does exactly what you want of "unlearning" facts that you've learned. Not sure if it helps in your particular instance, but it's very cool stuff and IIRC there is an implementation that extends datalog. https://en.wikipedia.org/wiki/Answer_set_programming
Re: I accidentally turned LLM memory into program analysis
#17This is the fact I’ve been struggling with for quite some time. It’s not because it forgets the facts, it’s because the invalidation doesn’t propagate.
My way of handling that is a decision log. For every project since I started doing that it’s working great. My CLAUDE.md instruct the agent to store my every decision to the file with a metadata when I made this decision and what was the context. The agent is using this file as an index of decisions and rarely lose a track. It also helps team members to find out more about the development phases.
Does your system invalidate the parts of the memory if these are not valid or relevant anymore or just store/retrieve?
Re: I accidentally turned LLM memory into program analysis
#18I reached a similar conclusion: LLMs should only really sit at the terminals of request fulfilment. 1. User request understanding: natural language -> a more rigorous representation, in my case Datalog. 2. Result interpretation: facts and derived facts -> natural language. Between those terminals, the work should be mechanical reasoning over some ontology or formal knowledge structure. That connects to another princi…
[dead]
Re: I accidentally turned LLM memory into program analysis
#19[flagged]
Re: I accidentally turned LLM memory into program analysis
#20This was a pretty awesome read, I liked it a lot!
What I found out during malware analysis is that LLM agents have a couple of quirks that you can solve by:
- optimize for short lived agent workflows
- use symbols as function contracts
- maintain decision and discovery state
- give LLMs CLI linters
- give LLMs access to knowledge bases
The linter part is mindblowing. I built linters that validate HTML or markdown or docx or Go or C files, for example, and they output what kind of structure is expected instead of useless token based errors (e.g. h4 inside h1? Must be h1 > h2 ...).
With linters the output quality of agents is just soo much better.
For program analysis, I'm currently exploring the idea of using an external ebpf daemon that programs can be observed with via a public API (which is the tool for the agent to use). Not sure if it'll do the trick yet, but I think it has lots of potential.
My stuff in case you're interested:
[1] https://github.com/cookiengineer/exocomp