Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use Prolog to improve LLM's reasoning

shchegrikovich.substack.com

Post reply on HN
Print view View original

121–130 of 158 posts

Re: Use Prolog to improve LLM's reasoning

#121

Post by samatman »

tannhaeuser wrote: post #84
jfengel wrote: post #36

Earlier quoted context omitted.

"Prolog" is like Lisp, a wide array of superficially similar languages that actually are quite diverse. Mind you, in that sense, Java and C# are more or less the same language, which has Prolog programmers nodding their heads and Java and C# developers screaming.

Nope. Prolog is an ISO-standardized language since 1995 and the spec was updated in 2012. Where older "legacy" Prolog implementations such as SWI, YAP, and SICStus are deviating from the standard is generally pretty well-known to Prolog practitioners, and the convener of ISO 13211 actually can verify claims of ISO conformance; for example, [1] is a link to the ISO certification of Quantum Prolog (the web app at [2]).…

You are referring to 'Prolog', replying to a post referring to '"Prolog"'.

Your constraint system is missing a clause ;-)

Re: Use Prolog to improve LLM's reasoning

#123

Post by upghost »

z5h wrote: post #117
javcasas wrote: post #97

Earlier quoted context omitted.

How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. Also DCGs for high level operations? Do you mean "use DCGs to parse strings that contain instructions" or do you parse things other than strings with DCGs? I'm assuming you take the parsed instructions and run them through some kind of interpreter that does the execution and audit trail.

> How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. I also sympathize. "false" as the default failure mode is a challenge with Prolog. Most Prologs I've used have good debugging/stepping features (see spy and trace predicates), logical debugging of pure monotonic Prolog can often help (explained by Markus Triska), you can easily write (use existing) m…

Ahh nice, I like this one!

Re: Use Prolog to improve LLM's reasoning

#124

Post by rdtsc »

upghost wrote: post #102
javcasas wrote: post #97

Earlier quoted context omitted.

How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. Also DCGs for high level operations? Do you mean "use DCGs to parse strings that contain instructions" or do you parse things other than strings with DCGs? I'm assuming you take the parsed instructions and run them through some kind of interpreter that does the execution and audit trail.

I sympathize. I nearly dropped Prolog for this reason until I learned about term_expansion/2 and goal_expansion/2. If what Prolog is doing you consider incorrect, _make it_ incorrect. DCGs can be used to convert any data structure to a sequence. Actually, they are capable of any graph to graph , so they could produce a sequence of commands. The oft-cited Markus Triska has some great work on this: https://youtu.be/vda…

I was watching some of the videos and in the intro Markus mentioned some of the most popular Prolog systems https://www.youtube.com/watch?v=8XUutFBbUrg

SICStus, GNU Prolog, XSB, Ciao, Scryer, Traella, Tau but does not mention at all of SWI-Prolog. I remember using SWI-Prolog back in the day, did it somehow fall out favor, or is there some animosity between implementation and Markus is just not in the SWI-Prolog camp?

Re: Use Prolog to improve LLM's reasoning

#125

Post by upghost »

z5h wrote: post #57
xelxebar wrote: post #46

Earlier quoted context omitted.

> over the past 2 years of heavy Prolog use Oh, cool. Mind if I pick your brain a bit? Recently, there was an HN post[0] of a paper that makes a case against pure logic languages in favor of "functional logic" ones, which they exhibit with Curry[1]. The setup argument is that Prolog's specs backtracking, which strongly downlimits it from full SLD resolution, causing fatally sharp edges in real world usage. Being fair…

So first, let's keep in mind that with no execution model, Prolog is still a "syntax" for Horn clauses. It's still a way to document knowledge. Add SLD resolution and we can compute. The paper (intentionally I presume) orders clauses of a simple predicate to illustrate (cause) a problem in Prolog. But what I actually find is the more time spent in Prolog, the more natural it is to express things in a way that is clea…

Strongly 2nd checking out Markus Triska's work. It's practically poetry.

It's really surprising to see someone at his level, a software reliability researcher no less, retain that level of passion about something. Usually folks with that level of experience are really grumpy and lose the evangelistic flair. I tend to maintain that level of evangelism about things I'm passionate for but no one would ever accuse me of being a "reliability researcher", lol.

In fact, I was/am a hardcore lisp hacker, and never in my life did I think I'd find something that would even come close to lisp for me. because it's FUN! but I'll be damned if Prolog isn't turning out to be even more fun in ways I didn't expect.

In fact the only thing I really miss about lisp at this point is structural editing (paredit and such).

And it's not like it's either/or, you can use both. But learning pure Prolog is really delightful if you follow the breadcrumbs (gold nuggets, honestly) Markus left.

If it doesn't kill you, that is!

Re: Use Prolog to improve LLM's reasoning

#126

Post by sgdfhijfgsdfgds »

sgdfhijfgsdfgds wrote: post #122

This is magical thinking. If an LLM can’t reason it isn’t going to be able to express itself clearly in Prolog. Suggesting otherwise is intellectually on the same level as trying to make up a small consistent per-sale loss with volume.

I know we're not supposed to comment on downvotes but I really question the logic of anyone who thinks that a thing that cannot reason can write a prolog program that is really going to be much more successful.

Prolog is actually pretty difficult to do right, even if you are skilled. It actually requires reasoning. You don't just write out facts and have the system do the work. And many of the examples in the training set will be wrong, naturally simplistic or be full of backtracking that is itself difficult for a person to comprehend at a glance; why should an LLM be better at it? There can't even be that much data in the training set.

Ultimately, though: stop believing in magical solutions to fundamental problems. This is nuts.

Re: Use Prolog to improve LLM's reasoning

#127

Post by YeGoblynQueenne »

upghost wrote: post #112
YeGoblynQueenne wrote: post #39

Earlier quoted context omitted.

I've programmed in Prolog for ~13 years and my PhD thesis is in machine learning of Prolog programs. How deep would you like me to go?

can you link your thesis? That sounds awesome!!

I can't seem to find it on the web. I don't think it's published online yet.

Re: Use Prolog to improve LLM's reasoning

#129

Post by shchegrikovich »

sgdfhijfgsdfgds wrote: post #126
sgdfhijfgsdfgds wrote: post #122

This is magical thinking. If an LLM can’t reason it isn’t going to be able to express itself clearly in Prolog. Suggesting otherwise is intellectually on the same level as trying to make up a small consistent per-sale loss with volume.

I know we're not supposed to comment on downvotes but I really question the logic of anyone who thinks that a thing that cannot reason can write a prolog program that is really going to be much more successful. Prolog is actually pretty difficult to do right, even if you are skilled. It actually requires reasoning. You don't just write out facts and have the system do the work. And many of the examples in the trainin…

I have another example - just a few people believed that you can apply 'a simple next token prediction algorithm' and achieve what we know as LLM. From my perspective, in the past few years, we've tried a lot of different approaches to improve LLM reasoning; some of them were good, others not so good. We need to keep trying and researching. 'Prolog + LLM' is not the answer to all questions, but it looks like a good step to move us forward.

Re: Use Prolog to improve LLM's reasoning

#130

Post by sgdfhijfgsdfgds »

Philpax wrote: post #24
YeGoblynQueenne wrote: post #22

That's not going to work. Garbage in - Garbage out is success-set equivalent to Garbage in - Prolog out. Garbage is garbage and failure to reason is failure to reason no matter the language. If your LLM can't translate your problem to a Prolog program that solves your problem- Prolog can't solve your problem.

This is a shallow critique that does not engage with the core idea. Specifying the problem is not the same as solving the problem.

It's actually pretty concise: Prolog isn't all that easy! That's why people don't use it.

Competent CS students fail Prolog courses all the time. A lot of Prolog on the internet will either be wrong, or it will be so riddled with unnecessary/unclear backtracking that an LLM won't be able to make more sense of it than it does words.

Post reply on HN