Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use Prolog to improve LLM's reasoning

shchegrikovich.substack.com

Post reply on HN
Print view View original

111–120 of 158 posts

Re: Use Prolog to improve LLM's reasoning

#111

Post by upghost »

hollerith wrote: post #66
z5h wrote: post #14

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…

What is your explanation for Prolog's lack of uptake so far? It has been around for 52 years and got a lot of buzz about 40 years ago because of its usefulness in creating expert systems.

Early systems were largely closed source, proprietary, licensed commercial systems. In some ways Prolog is 52 years old and in some ways it's 5 years old.

...I can relate...

Re: Use Prolog to improve LLM's reasoning

#112

Post by upghost »

YeGoblynQueenne wrote: post #39
Philpax wrote: post #24

Earlier quoted context omitted.

This is a shallow critique that does not engage with the core idea. Specifying the problem is not the same as solving the problem.

I've programmed in Prolog for ~13 years and my PhD thesis is in machine learning of Prolog programs. How deep would you like me to go?

can you link your thesis? That sounds awesome!!

Re: Use Prolog to improve LLM's reasoning

#113

Post by ianbicking »

gbanfalvi wrote: post #38
ianbicking wrote: post #35

I made a pipeline using Z3 (another prover language) to get LLMs to solve very specific puzzle problems: https://youtu.be/UjSf0rA1blc (and a presentation: https://youtu.be/TUAmfi8Ws1g ) Some thoughts: 1. Getting an LLM to model a problem accurately is a significant prompting exercise. Bridging casual logical statements and formal logic is difficult. E.g., "or" statements in English usually mean "xor" in logic. 2. Dom…

> 6. I'm not sure what tasks call for this kind of logical reasoning Basically any tasks that fulfill legal or business requirements? Both companies and governments are rushing to put LLMs into anything they can to avoid paying people. It’s vital to ascertain that, say, a benefits application is assessed properly and the LLM doesn’t hallucinate its way into an incorrect decision. I’d question if we really need LLMs i…

I think the best way to verify if a task could benefit from this is to try it by hand. What would a logical representation of legal requirements look like? If we can't model it by hand then we shouldn't expect an LLM to be able to do it either.

Probably 15 years ago I recall talking with someone at Columbia University who was working on an NLP project to help defendants understand the full implications of a plea deal. For instance if you plead guilty to a misdemeanor, what are all the implications elsewhere for someone who has that on their record. Or if you are on parole and plead guilty, the plea could have an effect on your parole.

The result might look a little more like search than logic. Like "rule 39.21 applies when (logical condition)". But then I can imagine growing the logical conditions... maybe you start with the most obvious formal definitions like crime severity, but then start to pull in other definitions as formal logical states as you see where that leads.

Re: Use Prolog to improve LLM's reasoning

#114

Post by YeGoblynQueenne »

tannhaeuser wrote: post #84
jfengel wrote: post #36

Earlier quoted context omitted.

"Prolog" is like Lisp, a wide array of superficially similar languages that actually are quite diverse. Mind you, in that sense, Java and C# are more or less the same language, which has Prolog programmers nodding their heads and Java and C# developers screaming.

Nope. Prolog is an ISO-standardized language since 1995 and the spec was updated in 2012. Where older "legacy" Prolog implementations such as SWI, YAP, and SICStus are deviating from the standard is generally pretty well-known to Prolog practitioners, and the convener of ISO 13211 actually can verify claims of ISO conformance; for example, [1] is a link to the ISO certification of Quantum Prolog (the web app at [2]).…

I think the OP may be referring to significantly divergent "Prologs" like Visual Prolog (a.k.a. Turbo Prolog) and Ciao.

Re: Use Prolog to improve LLM's reasoning

#115

Post by YeGoblynQueenne »

gorkempacaci wrote: post #31

The generated programs are only technically Prolog programs. They use CLPFD, which makes these constraint programs. Prolog programs are quite a bit more tricky with termination issues. I wouldn’t have nitpicked if it wasn’t in the title. Also, the experiment method has some flaws. Problems are hand-picked out of a random subset of the full set. Why not run the full set?

>> Why not run the full set?

Most likely cherry-picking. The approach is only going to work well in domains where Prolog is commonly used to write solutions to problems, like logical puzzles or constraint problems etc.

Re: Use Prolog to improve LLM's reasoning

#116

Post by YeGoblynQueenne »

luke_galea wrote: post #56

Super cool. I dig generating rules from within the LLM, but I'm not sure Prolog is the right choice in 2024. I love Prolog and had the opportunity to use it "in anger" years ago to handle temporal logic in a scheduling app. Great experience, but I've found that more modern rules engines like Drools (anything using the Rete algorithm) are a MUCH better fit for most use cases these days. If you are into this stuff, you…

Drools is a rules engine, but Prolog is a fully-fledged, general-purpose language, yes? For example SWI-Prolog has a bunch of http libraries and can be used as a web development language (using Prolog's clause database itself in place of some SQL). I don't think that'd be a sensible use case for Drools.

Re: Use Prolog to improve LLM's reasoning

#117

Post by z5h »

javcasas wrote: post #97
z5h wrote: post #58

Earlier quoted context omitted.

> What is some of your most hard-earned knowledge? 1. If you find yourself straying too often from coding in relations, and instead coding in instructive steps, you're going to end up with problems. 2. Use DCGs to create a DSL for any high level operations performed on data structures. The bi-directionality of Prolog's clauses means you can use this DSL to generate an audit trail of "commands executed" when Prolog so…

How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. Also DCGs for high level operations? Do you mean "use DCGs to parse strings that contain instructions" or do you parse things other than strings with DCGs? I'm assuming you take the parsed instructions and run them through some kind of interpreter that does the execution and audit trail.

> How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs.

I also sympathize. "false" as the default failure mode is a challenge with Prolog. Most Prologs I've used have good debugging/stepping features (see spy and trace predicates), logical debugging of pure monotonic Prolog can often help (explained by Markus Triska), you can easily write (use existing) meta predicates that assert a called predicate must not fail otherwise throw an exception. For example: here the ./ is supposed to look like a checkmark. So `./ true.` is true. `./ false` throws an exception.

  :- op(920, fy, './').
  :- meta_predicate './'(0).
  './'(X) :- call(X) *-> true ; prolog_debug:assertion_failed(fail, X).

Re: Use Prolog to improve LLM's reasoning

#118

Post by diggan »

YeGoblynQueenne wrote: post #110
diggan wrote: post #91

Earlier quoted context omitted.

With the right prompting you can get LLMs to output pretty much anything :) > What is the airspeed velocity of an unladen Eurasian blue tit? > I’m not sure. The specific airspeed velocity of an unladen Eurasian blue tit hasn't been studied or widely documented in the same way that birds like swallows have been. It would likely depend on many factors like the bird’s weight, wing shape, and wind conditions. If you’re l…

GPT-4o is a fan of Monty Python's O.o

Surprised me initially too, but then I remembered it's trained on web content, makes a lot of sense then.

Re: Use Prolog to improve LLM's reasoning

#119

Post by z5h »

YeGoblynQueenne wrote: post #104
z5h wrote: post #57

Earlier quoted context omitted.

So first, let's keep in mind that with no execution model, Prolog is still a "syntax" for Horn clauses. It's still a way to document knowledge. Add SLD resolution and we can compute. The paper (intentionally I presume) orders clauses of a simple predicate to illustrate (cause) a problem in Prolog. But what I actually find is the more time spent in Prolog, the more natural it is to express things in a way that is clea…

>> So I'd be concerned that Curry gives up the simplicity of Prolog's execution model, and ease of meta-programming. That was my concern with the paper listed above also. Functional syntax is to my mind needlessly over-complicated. Types are useful but you can roll your own if you need them, and Prolog makes that easy enough. >> I'm currently using SWI-Prolog. Performance is excellent, it has excellent high-level con…

> Hah! Hello fellow PCEmacs >> vim user :D

That made my day! Hello!

Re: Use Prolog to improve LLM's reasoning

#120

Post by tannhaeuser »

hollerith wrote: post #66
z5h wrote: post #14

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…

What is your explanation for Prolog's lack of uptake so far? It has been around for 52 years and got a lot of buzz about 40 years ago because of its usefulness in creating expert systems.

Not GP, but if Prolog has been around for 52 years, has many implementations old (SICStus/Quintus, SWI, yap, ciao, xsb, eclipse, many more) and new (Scryer, Quantum, Strawberry, Tau, Trealla, ...) then it follows it's not doing so bad after all.

I have to say the "expert system" quote comes from lack of insight and perhaps outdated CompSci lecture notes when even at the height of "expert systems" around 1990 or before, Prolog's backward-chaining was seen as the opposite of an expert system's forward-chaining.

It may have been the case that what was understood as an "AI language" was anathema following the AI crash around 1988/89 (the original "AI winter").

What also may have contributed to the impression of Prolog becoming less used around 2000-2012 or so is the effect of W3C's/TBL's "Semantic Web" and description logics efforts capturing a portion of the academic and commercial attention in graph and logic databases, as well as in other applied logic domains such as formal verification (in a quite direct sense considering research grants for OWL, etc.)

Post reply on HN