Earlier quoted context omitted.
This is a shallow critique that does not engage with the core idea. Specifying the problem is not the same as solving the problem.
I've programmed in Prolog for ~13 years and my PhD thesis is in machine learning of Prolog programs. How deep would you like me to go?
Use Prolog to improve LLM's reasoning
51–60 of 158 posts
Re: Use Prolog to improve LLM's reasoning
#52i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…
Is it your thought that for the average programmer Prolog is easier to read and maintain than say Go, C#, or Java?
I've played with and seriously used many languages in my career. My experience is that pure functional (done Elm style) is productive and scales well to a larger team. Dynamic stuff like Ruby/Javascript always has more bugs than you think, even with "full" test coverage. I'm not smart enough to make sense of my own Scheme meta-programming when I revisit it months later. I have loads (but dated) experience with Java and it (and peers) are relatively easy to read and maintain.
Prolog is very surprising, because it is homoiconic and immensely powerful in metaprogramming, BUT ... the declarative style and execution model reigns in the complexity/readability. A term is just a term. Nothing happens when you create a term. If/when a term is a goal, then you match it with the head of an existing predicate (something you've already coded). So it never gets too messy. Now, the biggest problem with Prolog is that it's so flexible, you'll perpetually be realizing that you could have coded something much more cleanly. So you do that, have less, code, it's nicer, etc. Doing this on a large team might not scale without effort.
Re: Use Prolog to improve LLM's reasoning
#53The generated programs are only technically Prolog programs. They use CLPFD, which makes these constraint programs. Prolog programs are quite a bit more tricky with termination issues. I wouldn’t have nitpicked if it wasn’t in the title. Also, the experiment method has some flaws. Problems are hand-picked out of a random subset of the full set. Why not run the full set?
Since you seem like an expert: is there a better technology for logical/constraint programming? I loved predicate calculus in school so it seems like there should be something out there for me, but so far no dice. This seems kinda related to the widely-discussed paradigm of “Linear Programming”, but I’ve also failed to find much of interest there behind all the talk of “Management Theory” and detailed mathematical efficiency comparisons.
I guess Curry (from above) might be the go-to these days?
Re: Use Prolog to improve LLM's reasoning
#54I tried an experiment with this using a Prolog interpreter with GPT-4 to try to answer complex logic questions. I found that it was really difficult because the model didn't seem to know Prolog well enough to write a description of any complexity. It seems like you used an interpreter in the loop which is likely to help. I'd also be interested to see how o1 would do in a task like this or if it even makes sense to us…
I bet one person could probably build a pretty good synthetic NL->Prolog dataset. ROI for paying that person would be high if you were building a foundation model (ie benefits beyond being able to output Prolog.)
Re: Use Prolog to improve LLM's reasoning
#55Earlier quoted context omitted.
Been shouting here and many places for quite a while that CoT and all similar stuff eventually leads to logic programming. So happy I’m not crazy.
You’re in good company — the most influential AI academic of all time, the cooky grandfather of AI who picked up right where (when!) Turing left off, the man hated by both camps yet somehow in charge of them, agrees with you. I’m talking about Marvin Minsky, of course. See: Logical vs. Analogical (Minsky, 1991) https://ojs.aaai.org/aimagazine/index.php/aimagazine/article... …the limitations of current machine intelli…
Re: Use Prolog to improve LLM's reasoning
#56I love Prolog and had the opportunity to use it "in anger" years ago to handle temporal logic in a scheduling app. Great experience, but I've found that more modern rules engines like Drools (anything using the Rete algorithm) are a MUCH better fit for most use cases these days.
If you are into this stuff, you might like the talk I gave on rules engines, prolog and how it led to erlang & elixir. https://www.youtube.com/watch?v=mDnntrhk-8g&t=1s
Re: Use Prolog to improve LLM's reasoning
#57i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…
> over the past 2 years of heavy Prolog use Oh, cool. Mind if I pick your brain a bit? Recently, there was an HN post[0] of a paper that makes a case against pure logic languages in favor of "functional logic" ones, which they exhibit with Curry[1]. The setup argument is that Prolog's specs backtracking, which strongly downlimits it from full SLD resolution, causing fatally sharp edges in real world usage. Being fair…
But what I actually find is the more time spent in Prolog, the more natural it is to express things in a way that is clear, logical and performant. As with any language/paradigm, there are a few gotchas to be experienced. But generally speaking, SLD resolution has never once been an obstacle (in the past 2 years) of coding.
The general execution model of Prolog is pretty simple. The lack of functions actually makes meta-programming much clearer and simpler. A term is just data, unless it's stated as a goal. It's only a valid goal if you've already defined its meaning.
So I'd be concerned that Curry gives up the simplicity of Prolog's execution model, and ease of meta-programming. I struggle with the lack of types in Prolog, but also know I can (at least in theory) use Prolog to solve correctness problems in Prolog code.
I'm currently using SWI-Prolog. Performance is excellent, it has excellent high-level concurrency primitives[0] (when was the last time you pegged all your cores solving a problem?), and many libraries. I might be one of the few people who has committed to using the integrated editor (PceEmacs) despite being a Vim person. PceEmacs is just too good at syntax highlighting and error detection.
At the same time, I'm a huge fan of Markus Triska. His Youtube[1] stuff is mind-expanding (watch all of it, even if you never write Prolog). He has an excellent book online[2]. I admire the way he explains and advances pure monotonic Prolog, and I appreciate the push for ISO conformance and his support for Prologs that that do the same (SWI is not on that list).
If you want to learn Prolog, watch all of Markus Triska's videos, read his book, and learn what Prolog could be in a perfect world. Then download SWI-Prolog, and maybe break some rules while getting things done at a blazing speed. Eventually you'll gravitate to what makes sense for you.
The Art of Prolog is a classic "must have". Clause and Effect is a good "hit the ground running" (on page 70 you're into symbolic differentiation via term rewriting).
0 https://www.swi-prolog.org/pldoc/man?section=thread
Re: Use Prolog to improve LLM's reasoning
#58i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…
> over the past 2 years of heavy Prolog use Oh, cool. Mind if I pick your brain a bit? Recently, there was an HN post[0] of a paper that makes a case against pure logic languages in favor of "functional logic" ones, which they exhibit with Curry[1]. The setup argument is that Prolog's specs backtracking, which strongly downlimits it from full SLD resolution, causing fatally sharp edges in real world usage. Being fair…
1. If you find yourself straying too often from coding in relations, and instead coding in instructive steps, you're going to end up with problems.
2. Use DCGs to create a DSL for any high level operations performed on data structures. The bi-directionality of Prolog's clauses means you can use this DSL to generate an audit trail of "commands executed" when Prolog solves a problem for you, but you can also use the audit trail and modify it to execute those commands on other data.