Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use Prolog to improve LLM's reasoning

shchegrikovich.substack.com

Post reply on HN
Print view View original

61–70 of 158 posts

Re: Use Prolog to improve LLM's reasoning

#61

Post by OutOfHere »

luke_galea wrote: post #56

Super cool. I dig generating rules from within the LLM, but I'm not sure Prolog is the right choice in 2024. I love Prolog and had the opportunity to use it "in anger" years ago to handle temporal logic in a scheduling app. Great experience, but I've found that more modern rules engines like Drools (anything using the Rete algorithm) are a MUCH better fit for most use cases these days. If you are into this stuff, you…

The choice is limited to the languages that LLMs already know really well. Fwiw, here is GPT's self-rating out of 10:

Python: 9, Prolog: 7, Datalog: 6, Mercury: 6, Curry: 5, Drools: 4

This is not even the full set of what the LLM might like to use. It may also like pyDatalog, SymPy, Haskell, Clingo ASP, ECLiPSe CLP, etc.

Re: Use Prolog to improve LLM's reasoning

#63

Post by OutOfHere »

YeGoblynQueenne wrote: post #22

That's not going to work. Garbage in - Garbage out is success-set equivalent to Garbage in - Prolog out. Garbage is garbage and failure to reason is failure to reason no matter the language. If your LLM can't translate your problem to a Prolog program that solves your problem- Prolog can't solve your problem.

Why do you think that the LLM cannot translate the problem into a program? Granted, it has been said that the Curry or Mercury languages may be better than Prolog at times with their functional logic programming features. Ultimately it's best if the LLM has the freedom to decide what's best to use for the problem from what it knows.

Re: Use Prolog to improve LLM's reasoning

#64

Post by int_19h »

bytebach wrote: post #30

An application I am developing for a customer needed to read constraints around clinical trials and essentially build a query from them. Constraints involve prior treatments, biomarkers, type of disease (cancers) etc. Using just an LLM did not produce reliable queries, despite trying many many prompts, so being an old Prolog hacker I wondered if using it might impose more 'logic' on the LLM. So we precede the textual…

I find that having GPT-4 write SQL queries to query the data source as needed to solve a complex task step-by-step also works pretty well (and you can give it the schema in form of CREATE TABLE). It's not exactly good at writing fast queries, but it can do some hella complex ones with nesting and joins to get exactly what it needs in one go.

Re: Use Prolog to improve LLM's reasoning

#65

Post by ianandrich »

larodi wrote: post #40
z5h wrote: post #14

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…

Been shouting here and many places for quite a while that CoT and all similar stuff eventually leads to logic programming. So happy I’m not crazy.

You are not crazy. Logic programming is the future

Re: Use Prolog to improve LLM's reasoning

#66

Post by hollerith »

z5h wrote: post #14

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…

What is your explanation for Prolog's lack of uptake so far?

It has been around for 52 years and got a lot of buzz about 40 years ago because of its usefulness in creating expert systems.

Re: Use Prolog to improve LLM's reasoning

#67

Post by dmichulke »

z5h wrote: post #14

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…

> It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know).

Would you be able to formulate all those "caveats and exceptions" in Prolog?

Re: Use Prolog to improve LLM's reasoning

#68

Post by aitchnyu »

DeborahWrites wrote: post #26

You're telling me the seemingly arbitrary 6 weeks of Prolog on my comp sci course 11yrs ago is suddenly about to be relevant? I did not see this one coming . . .

I rolled my eyes when iterating a list meant splitting it into first and rest and recursing on the rest. I would do side projects in Scala a few years later of my own interest.

Re: Use Prolog to improve LLM's reasoning

#69

Post by microtonal »

z5h wrote: post #57
xelxebar wrote: post #46

Earlier quoted context omitted.

> over the past 2 years of heavy Prolog use Oh, cool. Mind if I pick your brain a bit? Recently, there was an HN post[0] of a paper that makes a case against pure logic languages in favor of "functional logic" ones, which they exhibit with Curry[1]. The setup argument is that Prolog's specs backtracking, which strongly downlimits it from full SLD resolution, causing fatally sharp edges in real world usage. Being fair…

So first, let's keep in mind that with no execution model, Prolog is still a "syntax" for Horn clauses. It's still a way to document knowledge. Add SLD resolution and we can compute. The paper (intentionally I presume) orders clauses of a simple predicate to illustrate (cause) a problem in Prolog. But what I actually find is the more time spent in Prolog, the more natural it is to express things in a way that is clea…

I'm currently using SWI-Prolog.

Still checking every now and then if SICStus has open sourced. I used Prolog daily during my PhD and SICStus had such nice features. E.g. it could raise an exception when no more heap space can be allocated or when a 'call' would not finish within a given time. These features made it much easier to use Prolog in real-world systems (this was a parsing system and when parsing a very large corpus, this was highly preferable over simply crashing the interpreter).

Maybe things have changed, but this wasn't possible with SWI at the time. Even worse, most C extensions would use malloc directly, making it impossible to track allocations done by extensions.

Re: Use Prolog to improve LLM's reasoning

#70

Post by gorkempacaci »

bbor wrote: post #53
gorkempacaci wrote: post #31

The generated programs are only technically Prolog programs. They use CLPFD, which makes these constraint programs. Prolog programs are quite a bit more tricky with termination issues. I wouldn’t have nitpicked if it wasn’t in the title. Also, the experiment method has some flaws. Problems are hand-picked out of a random subset of the full set. Why not run the full set?

Yeah I’m a huge proponent of this general philosophy, but after being introduced to prolog itself for a third of a semester back in undergrad I decided to stay far, far away. The vision never quite came through as clearly as it did for the other wacky languages, namely the functional family (Lisp and Haskell in my case). I believe you on the fundamental termination issues, but just basic phrasing seemed unnecessarily…

Curious to know what part of syntax you found convoluted. If you remember any examples I’d appreciate it. Maybe you want a constraint programming environment instead. As example check out Conjure from St Andrews: https://conjure.readthedocs.io/en/latest/tutorials-notebook....

More generally there are the theorem provers like Coq, etc., but their use cases are even more specific.

Post reply on HN