Live data from Hacker News

Use Prolog to improve LLM's reasoning

shchegrikovich.substack.com

71–80 of 158 posts

Re: Use Prolog to improve LLM's reasoning

#71
post #14

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declar…

Have you tried - https://www.minizinc.org/?

Re: Use Prolog to improve LLM's reasoning

#72
post #55
post #48

Earlier quoted context omitted.

You’re in good company — the most influential AI academic of all time, the cooky grandfather of AI who picked up right where (when!) Turing left off, the man hated by both camps yet somehow in charge of them, agrees with you. I’m talking about Marvin Minsky, of course. See: Logical vs. Analogical (Minsky, 1991) https://ojs.aaai.org/aimagazine/index.php/aimagazine/article... …the limitations of current machine intelli…

Just throwing this out there for someone, "Scruffier Kind of Machinery" is a good name for a book, company or band.

Heh, well spotted. SSKOM will be overtaking KGATLW any day now…

Re: Use Prolog to improve LLM's reasoning

#73
post #13
post #7

I tried an experiment with this using a Prolog interpreter with GPT-4 to try to answer complex logic questions. I found that it was really difficult because the model didn't seem to know Prolog well enough to write a description of any complexity. It seems like you used an interpreter in the loop which is likely to help. I'd also be interested to see how o1 would do in a task like this or if it even makes sense to us…

I also wrote wrote an LLM to Prolog interpreter for a hackathon called "Logical". With a few hours effort I'm sure it could be improved. https://github.com/Hendler/logical I think while LLMs may approach completeness here, it's good to have an interpretable system to audit/verify and reproduce results.

This is really cool!

Re: Use Prolog to improve LLM's reasoning

#74

I made a pipeline using Z3 (another prover language) to get LLMs to solve very specific puzzle problems: https://youtu.be/UjSf0rA1blc (and a presentation: https://youtu.be/TUAmfi8Ws1g ) Some thoughts: 1. Getting an LLM to model a problem accurately is a significant prompting exercise. Bridging casual logical statements and formal logic is difficult. E.g., "or" statements in English usually mean "xor" in logic. 2. Dom…

I think another problem which we can try to solve with this approach is factual verification.

Re: Use Prolog to improve LLM's reasoning

#75

Would Creating a prolog dataset would be beneficial, so that future LLM's can be trained on it and then they would be able to output prolog code.

I discussed this a few weeks back. The idea is to take a Python dataset, as Python is the most popular language, and write a transpiler to Prolog with the help of llms. So, creating this synthetic dataset is not a huge problem.

Re: Use Prolog to improve LLM's reasoning

#77

An application I am developing for a customer needed to read constraints around clinical trials and essentially build a query from them. Constraints involve prior treatments, biomarkers, type of disease (cancers) etc. Using just an LLM did not produce reliable queries, despite trying many many prompts, so being an old Prolog hacker I wondered if using it might impose more 'logic' on the LLM. So we precede the textual…

I agree; we can do much more with LLMs if we focus on declarative-style languages.

Re: Use Prolog to improve LLM's reasoning

#78

Does anyone know why US attorneys and law firms are not using Prolog-based apps to automate the low-hanging fruit of issue-spotting?

Or something like Catala language - https://catala-lang.org/? Catala: A Programming Language for the Law - https://arxiv.org/abs/2103.03198

Re: Use Prolog to improve LLM's reasoning

#79

An application I am developing for a customer needed to read constraints around clinical trials and essentially build a query from them. Constraints involve prior treatments, biomarkers, type of disease (cancers) etc. Using just an LLM did not produce reliable queries, despite trying many many prompts, so being an old Prolog hacker I wondered if using it might impose more 'logic' on the LLM. So we precede the textual…

This sounds like a very important observation.

Re: Use Prolog to improve LLM's reasoning

#80
I implemented Prolog and Z3 as a function tool for my little OpenAI Assistants API client. Z3 (SMT-LIB, actually) seemed even more promising. The model speaks both languages. However, I didn't find any convincing use-cases yet. But it seems a logical extension of the idea to provide a programming language via a function tool to solve problems. So you start thinking "What else has text input and output, and a powerful engine in between?"
Post reply on HN