Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use Prolog to improve LLM's reasoning

shchegrikovich.substack.com

Post reply on HN
Print view View original

101–110 of 158 posts

Re: Use Prolog to improve LLM's reasoning

#101

Post by YeGoblynQueenne »

javcasas wrote: post #97
z5h wrote: post #58

Earlier quoted context omitted.

> What is some of your most hard-earned knowledge? 1. If you find yourself straying too often from coding in relations, and instead coding in instructive steps, you're going to end up with problems. 2. Use DCGs to create a DSL for any high level operations performed on data structures. The bi-directionality of Prolog's clauses means you can use this DSL to generate an audit trail of "commands executed" when Prolog so…

How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. Also DCGs for high level operations? Do you mean "use DCGs to parse strings that contain instructions" or do you parse things other than strings with DCGs? I'm assuming you take the parsed instructions and run them through some kind of interpreter that does the execution and audit trail.

(Not OP)

>> How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs.

You need to include exception handling in your DCG rules. For example, in Prolog-like pseudocode:

  pink_apples([A|As]) --> [A], { red_apple(A), throw(error(type_error(pink_apple,red_apple),_)) }.

  % Raises type error:
  ERROR: Type error: `pink_apple' expected, found `red_apple' (an atom)
  ERROR: In:
  ERROR:   [12] throw(error(type_error(pink_apple,red_apple),_10070))
Called from a source file the error output will list the line in the source file where the exception was raised. There are more tools to debug the error:

https://www.swi-prolog.org/pldoc/man?section=exception

DCGs parse lists, not strings, as such. So the input can be anything you can put in the form of a list.

Re: Use Prolog to improve LLM's reasoning

#102

Post by upghost »

javcasas wrote: post #97
z5h wrote: post #58

Earlier quoted context omitted.

> What is some of your most hard-earned knowledge? 1. If you find yourself straying too often from coding in relations, and instead coding in instructive steps, you're going to end up with problems. 2. Use DCGs to create a DSL for any high level operations performed on data structures. The bi-directionality of Prolog's clauses means you can use this DSL to generate an audit trail of "commands executed" when Prolog so…

How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. Also DCGs for high level operations? Do you mean "use DCGs to parse strings that contain instructions" or do you parse things other than strings with DCGs? I'm assuming you take the parsed instructions and run them through some kind of interpreter that does the execution and audit trail.

I sympathize. I nearly dropped Prolog for this reason until I learned about term_expansion/2 and goal_expansion/2.

If what Prolog is doing you consider incorrect, _make it_ incorrect.

DCGs can be used to convert any data structure to a sequence. Actually, they are capable of any graph to graph , so they could produce a sequence of commands.

The oft-cited Markus Triska has some great work on this:

  https://youtu.be/vdabv9EkYrY?feature=shared

  https://www.metalevel.at/zurg/
You can also use Prolog as a coordinator on external systems.

See https://github.com/mthom/scryer-prolog/blob/master/src/lib/p...

Re: Use Prolog to improve LLM's reasoning

#103

Post by upghost »

YeGoblynQueenne wrote: post #101
javcasas wrote: post #97

Earlier quoted context omitted.

How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. Also DCGs for high level operations? Do you mean "use DCGs to parse strings that contain instructions" or do you parse things other than strings with DCGs? I'm assuming you take the parsed instructions and run them through some kind of interpreter that does the execution and audit trail.

(Not OP) >> How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. You need to include exception handling in your DCG rules. For example, in Prolog-like pseudocode: pink_apples([A|As]) --> [A], { red_apple(A), throw(error(type_error(pink_apple,red_apple),_)) }. % Raises type error: ERROR: Type error: `pink_apple' expected, found `red_apple' (an atom) ERROR…

And this is the person who saved me from nearly dropping Prolog!!

Re: Use Prolog to improve LLM's reasoning

#104

Post by YeGoblynQueenne »

z5h wrote: post #57
xelxebar wrote: post #46

Earlier quoted context omitted.

> over the past 2 years of heavy Prolog use Oh, cool. Mind if I pick your brain a bit? Recently, there was an HN post[0] of a paper that makes a case against pure logic languages in favor of "functional logic" ones, which they exhibit with Curry[1]. The setup argument is that Prolog's specs backtracking, which strongly downlimits it from full SLD resolution, causing fatally sharp edges in real world usage. Being fair…

So first, let's keep in mind that with no execution model, Prolog is still a "syntax" for Horn clauses. It's still a way to document knowledge. Add SLD resolution and we can compute. The paper (intentionally I presume) orders clauses of a simple predicate to illustrate (cause) a problem in Prolog. But what I actually find is the more time spent in Prolog, the more natural it is to express things in a way that is clea…

>> So I'd be concerned that Curry gives up the simplicity of Prolog's execution model, and ease of meta-programming.

That was my concern with the paper listed above also. Functional syntax is to my mind needlessly over-complicated. Types are useful but you can roll your own if you need them, and Prolog makes that easy enough.

>> I'm currently using SWI-Prolog. Performance is excellent, it has excellent high-level concurrency primitives[0] (when was the last time you pegged all your cores solving a problem?), and many libraries. I might be one of the few people who has committed to using the integrated editor (PceEmacs) despite being a Vim person. PceEmacs is just too good at syntax highlighting and error detection.

Hah! Hello fellow PCEmacs >> vim user :D

Markus Triska's stuff is good and he's undeniably an expert in Prolog programming, I mean duh, but he and a couple of others have needlessly caused a rift in the Prolog community (mainly between themselves and everyone else) by being so stroppy about the ISO non-conformance of SWI-Prolog. I hope Markus is reading this. The Prolog community is very small and dwindling and we can't afford such drama. We need more Prolog compilers, yes, ISO conformance is good, yes, but SWI-Prolog is a robust and battle-tested implementation and it is the ISO standard that should be leaning on its experience of being a real-world Prolog used by real programmers and not just the other way around.

Re: Use Prolog to improve LLM's reasoning

#105

Post by YeGoblynQueenne »

microtonal wrote: post #69
z5h wrote: post #57

Earlier quoted context omitted.

So first, let's keep in mind that with no execution model, Prolog is still a "syntax" for Horn clauses. It's still a way to document knowledge. Add SLD resolution and we can compute. The paper (intentionally I presume) orders clauses of a simple predicate to illustrate (cause) a problem in Prolog. But what I actually find is the more time spent in Prolog, the more natural it is to express things in a way that is clea…

I'm currently using SWI-Prolog. Still checking every now and then if SICStus has open sourced. I used Prolog daily during my PhD and SICStus had such nice features. E.g. it could raise an exception when no more heap space can be allocated or when a 'call' would not finish within a given time. These features made it much easier to use Prolog in real-world systems (this was a parsing system and when parsing a very larg…

>> Still checking every now and then if SICStus has open sourced.

There used to be a free (though proprietary) version of Quintus, the predecessor of SICStus. It might still be around somewhere if you look hard enough.

Re: Use Prolog to improve LLM's reasoning

#106

Post by YeGoblynQueenne »

upghost wrote: post #103
YeGoblynQueenne wrote: post #101

Earlier quoted context omitted.

(Not OP) >> How do you debug DCGs? I get "false." instead of "syntax error at line 23", which is unacceptable for bigger inputs. You need to include exception handling in your DCG rules. For example, in Prolog-like pseudocode: pink_apples([A|As]) --> [A], { red_apple(A), throw(error(type_error(pink_apple,red_apple),_)) }. % Raises type error: ERROR: Type error: `pink_apple' expected, found `red_apple' (an atom) ERROR…

And this is the person who saved me from nearly dropping Prolog!!

Hope you don't end up cursing me down the line :P

Re: Use Prolog to improve LLM's reasoning

#107

Post by YeGoblynQueenne »

bbor wrote: post #48
larodi wrote: post #40

Earlier quoted context omitted.

Been shouting here and many places for quite a while that CoT and all similar stuff eventually leads to logic programming. So happy I’m not crazy.

You’re in good company — the most influential AI academic of all time, the cooky grandfather of AI who picked up right where (when!) Turing left off, the man hated by both camps yet somehow in charge of them, agrees with you. I’m talking about Marvin Minsky, of course. See: Logical vs. Analogical (Minsky, 1991) https://ojs.aaai.org/aimagazine/index.php/aimagazine/article... …the limitations of current machine intelli…

Minsky is right. The "rift" between symbolic and sub-symbolic, or learning and reasoning has only impeded progress. The problem is it's very hard to be an expert on both at once, and it's getting harder and harder as more and more work is done on both.

Since you mentioned Hinton, he has worked hard to entrench the idea that symbolic AI failed. I was listening to a lecture he gave [1] and he went on and On and ON about how symbolic AI was a stupid idea and it was conclusively proved wrong by neural nets. He went on for so long bashing symbolic AI to the ground that at some point I started wondering whether he's deep down worried that there might be a resurgence of it right on time to address the limitations of neural nets with respect to reasoning, before he and his mates have the chance to figure out how to overcome it. Which, well, good luck with that.

_________________

[1] https://youtu.be/N1TEjTeQeg0?si=OIYY3wnDbi7rzJt3

Re: Use Prolog to improve LLM's reasoning

#108

Post by YeGoblynQueenne »

z5h wrote: post #52
tomcam wrote: post #32

Earlier quoted context omitted.

Is it your thought that for the average programmer Prolog is easier to read and maintain than say Go, C#, or Java?

I'm surprised at how readable Prolog is. I've played with and seriously used many languages in my career. My experience is that pure functional (done Elm style) is productive and scales well to a larger team. Dynamic stuff like Ruby/Javascript always has more bugs than you think, even with "full" test coverage. I'm not smart enough to make sense of my own Scheme meta-programming when I revisit it months later. I have…

>> I'm not smart enough to make sense of my own Scheme meta-programming when I revisit it months later.

Then be smart enough to comment your code :P

>> Prolog is very surprising, because it is homoiconic and immensely powerful in metaprogramming, BUT ... the declarative style and execution model reigns in the complexity/readability.

Iiiish? This is from one of my yesterday's commit messages:

    * New look and look-around actions in the Basic Sim Environment allow
      for looking ahead in eight direction. This does get a liiittle bit
      complicated, or rather there's the usual millefeuille of abstraction
      layers on top of abstraction layers all the way down pou that mou ta
      spasei kapoia stigmi but OK.
The Greek-lish interjection says approximately "that is going to bite me in the ass down the line". Because it will. The better I get with Prolog the more I worry nobody will be able to maintain my code but myself, and my future self will hate me with deep, burning passion.

Re: Use Prolog to improve LLM's reasoning

#109

Post by YeGoblynQueenne »

pjmlp wrote: post #81
tomcam wrote: post #32

Earlier quoted context omitted.

Is it your thought that for the average programmer Prolog is easier to read and maintain than say Go, C#, or Java?

As someone that went through a degree where Prolog and LP was cherisched, I would say yes, however LP might be even weirder to start into than even FP. Many folks on our degree couldn't be happier when they didn't had to see Prolog ever again, while me and others went on to take our chances on the national LP challenge across universities. Tarski's World was a good way back then to dive into what LP is all about, wit…

Out of curiosity, in which university did you study for your degree?

I did my MSc in Sussex Uni which was one of the centers in the UK where logic programming was developed, but when I got there in 2014 there was no trace of that history. From conversations with professors and past students it seems that Sussex tried to ram Prolog hard down students' throats and that caused a furious backlash so that nobody wanted to hear about it anymore after the '90s to early 2000's.

Re: Use Prolog to improve LLM's reasoning

#110

Post by YeGoblynQueenne »

diggan wrote: post #91
exe34 wrote: post #83

Earlier quoted context omitted.

> top of a Prolog-derived language who's name escapes me are you saying we've made a huge leap in LLMs - that they can now admit when they don't know something?

With the right prompting you can get LLMs to output pretty much anything :) > What is the airspeed velocity of an unladen Eurasian blue tit? > I’m not sure. The specific airspeed velocity of an unladen Eurasian blue tit hasn't been studied or widely documented in the same way that birds like swallows have been. It would likely depend on many factors like the bird’s weight, wing shape, and wind conditions. If you’re l…

GPT-4o is a fan of Monty Python's O.o
Post reply on HN