Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models as compilers: Simulating pseudocode execution

arxiv.org

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Language models as compilers: Simulating pseudocode execution

#31

Post by DebtDeflation »

pkoird wrote: post #5

Any sufficiently advanced LLM is indistinguishable from Prolog. I half-jest but I envision the direction of LLM research to head towards a parser-oriented setup where LLMs merely extract the entities and relations and the actual logic is done by a logical engine such as Prolog.

>where LLMs merely extract the entities and relations and the actual logic is done by a logical engine such as Prolog.

Graph RAG is an emerging design pattern where factual information is retrieved from knowledge graphs to augment the textual information retrieved from document stores in classic RAG before submitting to an LLM. What you are proposing is essentially using the LLM to build the knowledge graph in the first place. Would be interesting to see the two techniques combined along with some sort of planner/optimizer in the middle.

Re: Language models as compilers: Simulating pseudocode execution

#32

Post by astrange »

pkoird wrote: post #5

Any sufficiently advanced LLM is indistinguishable from Prolog. I half-jest but I envision the direction of LLM research to head towards a parser-oriented setup where LLMs merely extract the entities and relations and the actual logic is done by a logical engine such as Prolog.

The problem is that Prolog is already "AI" as in expert system GOFAI - that approach already failed, so you shouldn't believe people claiming that an entirely different connectionist approach is going to prove that their failed alternative worked all along.

Re: Language models as compilers: Simulating pseudocode execution

#33

Post by thesz »

ogogmad wrote: post #26
thesz wrote: post #18

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Cyc#MathCraft Quote: One Cyc application aims to help students doing math at a 6th grade level, helping them much more deeply understand that subject matter... Unlike almost all other educational software, where the computer plays the role of the teacher, this application of Cyc, called MathCraft, has Cyc play the role of a fellow student who is always slightly more confused than you, th…

> I suspect it is still hard for LLMs I just gave it to Claude: https://imgur.com/a/fQQOy1d

MathCraft [1] is little more [2] than chat.

  [1] https://www.cyc.com/mathcraft/
  [2] https://www.youtube.com/watch?v=pbrp7MzBDm0

Re: Language models as compilers: Simulating pseudocode execution

#34

Post by tiborsaas »

Voultapher wrote: post #19

Non deterministic compilers, yay! Where do I sign up? In more seriousness, miscompilations or in general unexpected behavior caused by layers below you are expensive to find and fix. I think LLMs have a long way to go before such use cases seem appealing to me.

It's better to have a non-deterministic compiler for a task that would be really hard to write an algorithm for otherwise.

Re: Language models as compilers: Simulating pseudocode execution

#35

Post by eeue56 »

I wrote a toy language along these lines a while back[0]. Basically, types and function signatures, with comments in English, produce a valid program. You write a type and a comment, and the compiler goes through GPT to run the code. Fun novel idea.

[0] - https://github.com/eeue56/neuro-lingo

Re: Language models as compilers: Simulating pseudocode execution

#36

Post by Phillipharryt »

ginko wrote: post #30
Voultapher wrote: post #19

Non deterministic compilers, yay! Where do I sign up? In more seriousness, miscompilations or in general unexpected behavior caused by layers below you are expensive to find and fix. I think LLMs have a long way to go before such use cases seem appealing to me.

Even regular compilers need quite a bit of nudging to give deterministic results.

Correct me if I'm wrong here, but I am under the impression they're only non-deterministic in the practical sense (i.e, it produces this output on my machine, I can't know what minute differences there are on your machine), but that's not non-deterministic in the truest sense. If you have completely identical inputs you will get the exact same output, ergo, deterministic.

Re: Language models as compilers: Simulating pseudocode execution

#37

Post by codesnik »

astrange wrote: post #32
pkoird wrote: post #5

Any sufficiently advanced LLM is indistinguishable from Prolog. I half-jest but I envision the direction of LLM research to head towards a parser-oriented setup where LLMs merely extract the entities and relations and the actual logic is done by a logical engine such as Prolog.

The problem is that Prolog is already "AI" as in expert system GOFAI - that approach already failed, so you shouldn't believe people claiming that an entirely different connectionist approach is going to prove that their failed alternative worked all along.

maybe that's because prolog from 80ies basically operated in 1bit space and had just 1000s of parameters, crafted by hand. But nowadays...

Re: Language models as compilers: Simulating pseudocode execution

#38

Post by novideogame »

I think the title is a little misleading. The main difference between this paper and CoC (Chain of Code) is that the LLM is instructed to make a plan to solve all the given instances and then code that plan in pseudocode, while in CoC the plan is to solve the single instance given.

From the paper: The main difference between THINK-AND-EXECUTE and CoC is that we use pseu- docodes which are generated to express logic shared among the tasks instances, while CoC incorporates pseudocode as part of the intermediate reasoning steps towards the solution of a given instance. Hence, the results indicate the advantages of applying pseudocode for the generation of task-level instruction over solely using them as a part of rationales.

I find the phrase "as a part of rationales" a little strange, but English is not my native language.

Re: Language models as compilers: Simulating pseudocode execution

#39

Post by ingigauti »

Couple of weeks ago I published a new programming language called Plang (as in pseudo language) that uses LLM to translate user intent into executable code, basically LLM as a compiler.

It saves you incredible amount of work, cutting code writing down by 90%+. The built code is deterministic(it will never change after build) and as a programmer you can validate the code that will be executed. It compiles to C#, so it handles GC, encoding, etc. that languages need to solve, so I can focus on other areas.

Plang also has some features that other language don't have, e.g. events on variables, built in identity and interesting(I think) approach to privacy.

I have not been advertising to much since it is still early development and I create still to many breaking changes, but help is welcome(and needed) so if it something that is interesting to you the repo is at https://github.com/plangHQ

Re: Language models as compilers: Simulating pseudocode execution

#40

Post by taneq »

Voultapher wrote: post #19

Non deterministic compilers, yay! Where do I sign up? In more seriousness, miscompilations or in general unexpected behavior caused by layers below you are expensive to find and fix. I think LLMs have a long way to go before such use cases seem appealing to me.

Non-determinism is an implementation detail, not an intrinsic property, as I understand it (at least as long as you're setting temperature to zero).

I likewise don't really think LLMs are the right tool for this job, though. There's a whole class of systems that we built because humans take a long time to learn new skills, are fallible and non-repeatable, and get bored easily. Compilers are in this group along with sewing machines, CNC machines, automatic gearboxes, and design rules checking in CAD.

Maybe they could provide heuristics for optimising compilers with the output run through a formal verification check afterwards?

Post reply on HN