Live data from Hacker News

I accidentally turned LLM memory into program analysis

pwning.systems

21–30 of 94 posts

Re: I accidentally turned LLM memory into program analysis

#21
post #5

Very cool. I recall an HN submission (which I can't find offhand unfortunately) that did something similar -- it used an LLM to decompose articles into a set of statements which were used to construct an entity-relationship graph of facts and events. It then queried that using conventional graph query methods, much like DataLog / Lemmalog is doing here. I remember it was particularly effective at answering timeline-b…

Is it this one:

https://news.ycombinator.com/item?id=41445445

Re: I accidentally turned LLM memory into program analysis

#24

I reached a similar conclusion: LLMs should only really sit at the terminals of request fulfilment. 1. User request understanding: natural language -> a more rigorous representation, in my case Datalog. 2. Result interpretation: facts and derived facts -> natural language. Between those terminals, the work should be mechanical reasoning over some ontology or formal knowledge structure. That connects to another princi…

Bayesian posteriors in the wild. Love it!

Re: I accidentally turned LLM memory into program analysis

#26
So he's using an LLM to generate data stored in an "is_a" representation. That's so classic AI.

Soon, he'll discover that he needs quantifiers. Then that "for all" is too strong sometimes, and he needs "for most". That way lies Cyc.

It's not a bad idea. But it does have a history.

Re: I accidentally turned LLM memory into program analysis

#28
Great work! If anyone is looking for a way to integrate something like this into their own harness or the pi coding agent, then you might be interested in DeepClause [0]. It comes with a Prolog-like language implemented on top of SWI-Prolog (WASM Version). The purpose of the project is to allow for broad experimentation around the intersection of LLMs/Agents and GOFAI. So you could use it to build memory systems like OP did, create executable specs, define graphs and loops for agents and subagents... It also comes with a pi extension that greatly simplifies getting started with it.

Opposed to OP, DeepClause uses Prolog semantics, so running some more complex queries on knowledgebases might cause some issues (which is the use case where a Datalog might be more useful). For smaller scales it should be fine though.

[0] https://github.com/deepclause/deepclause-sdk [1] https://github.com/deepclause/deepclause-pi

Re: I accidentally turned LLM memory into program analysis

#30

I reached a similar conclusion: LLMs should only really sit at the terminals of request fulfilment. 1. User request understanding: natural language -> a more rigorous representation, in my case Datalog. 2. Result interpretation: facts and derived facts -> natural language. Between those terminals, the work should be mechanical reasoning over some ontology or formal knowledge structure. That connects to another princi…

Theoretically but practically any LLM generated infra/classification set is going to drift due to inaccuracy and harm IR/whatever logical process you're using. I am a big fan of using a loose taxonomy but it's not been revolutionary.
Post reply on HN