Live data from Hacker News

Language models as compilers: Simulating pseudocode execution

arxiv.org

41–50 of 57 posts

Re: Language models as compilers: Simulating pseudocode execution

#41

Couple of weeks ago I published a new programming language called Plang (as in pseudo language) that uses LLM to translate user intent into executable code, basically LLM as a compiler. It saves you incredible amount of work, cutting code writing down by 90%+. The built code is deterministic(it will never change after build) and as a programmer you can validate the code that will be executed. It compiles to C#, so it…

In my experience it’s not exactly trivial to validate code you didn’t write yourself, because you have to think through it in similar depth to when you write it yourself. While on the one hand you save the time of coming up with a solution, the task of merely verifying an existing solution is also more tedious, because it isn’t intermixed with the problem-solving activity that you perform when writing the code yourself. There is an increased risk to fall for code that looks correct at first blush but is still subtly wrong, because you didn’t spend time iterating on it. It doesn’t seem plausible to me that you would save 90% of the work, unless it’s boiler plate-heavy code that requires only little analytical though.

Re: Language models as compilers: Simulating pseudocode execution

#42
post #30

Earlier quoted context omitted.

Even regular compilers need quite a bit of nudging to give deterministic results.

Correct me if I'm wrong here, but I am under the impression they're only non-deterministic in the practical sense (i.e, it produces this output on my machine, I can't know what minute differences there are on your machine), but that's not non-deterministic in the truest sense. If you have completely identical inputs you will get the exact same output, ergo, deterministic.

You are correct. Compilers are deterministic, but reproducible builds can be a challenge.

Re: Language models as compilers: Simulating pseudocode execution

#43
post #10

If you train a LLM to compile, you probably also want to set the randomness to zero, if that is the case you’ve just “brute forced” an actual compiler

you don't want a creative compiler?

I see no inherent conflict between creativity and determinism. You could give the compiler a seed value to perturbate the solution space.

Re: Language models as compilers: Simulating pseudocode execution

#44
post #41

Couple of weeks ago I published a new programming language called Plang (as in pseudo language) that uses LLM to translate user intent into executable code, basically LLM as a compiler. It saves you incredible amount of work, cutting code writing down by 90%+. The built code is deterministic(it will never change after build) and as a programmer you can validate the code that will be executed. It compiles to C#, so it…

In my experience it’s not exactly trivial to validate code you didn’t write yourself, because you have to think through it in similar depth to when you write it yourself. While on the one hand you save the time of coming up with a solution, the task of merely verifying an existing solution is also more tedious, because it isn’t intermixed with the problem-solving activity that you perform when writing the code yourse…

I agree with you. I never liked how AI is really generating ton of code for us, then you need to read through it and understand it. Plus, the fail rate is to high.

That is why I design the language the way it is. You must define each step you want to happen in your application. Lets take for example user registration, it looks like this

--- plang code ---

CreateUser

- Make sure %password% and %email% is not empty

- Hash %password%, write to %hashedPassword%

- Insert into users, %hashedPassword%, %email%

- Post, create user in MailChimp, Bearer:%Settings.MailChimpApi%, %email%

- Create bearer token from %email%, write to %bearer%

- Write %bearer% to web response

--- plang code ---

That is an executable code in plang. It's easy to read through and understand. You need to have domain knowledge, such as what is hashing and bearer token. You are still programming, just at higher level.

Validating what will execute, you need to learn, just like with any language, but it is relatively simple and you start to trust the result with time(at least I have)

Compared to the 130 lines or so of code in C# for the same logic, https://gist.github.com/ingig/491ac9b13d65f40cc24ee5aed0408b... That´s about 95% reduction of code, and I see this repeatedly.

Re: Language models as compilers: Simulating pseudocode execution

#45

Non deterministic compilers, yay! Where do I sign up? In more seriousness, miscompilations or in general unexpected behavior caused by layers below you are expensive to find and fix. I think LLMs have a long way to go before such use cases seem appealing to me.

It's better to have a non-deterministic compiler for a task that would be really hard to write an algorithm for otherwise.

But are LLMs really better at algorithm writing than you? I’ve found that they work best when I’ve already pseudocoded the algorithm.

Re: Language models as compilers: Simulating pseudocode execution

#46

Non deterministic compilers, yay! Where do I sign up? In more seriousness, miscompilations or in general unexpected behavior caused by layers below you are expensive to find and fix. I think LLMs have a long way to go before such use cases seem appealing to me.

I (kinda) solved this with neuro-lingo[0] with the concept of pinning. Basically, once you have a version of a function implementation that works, you can pin it and it won't be regenerated when it's "compiled". The alternative approach would be to have tests be the only code a developer writes, and then make LLMs generate code to match the implementation for those, running the tests to ensure it's valid.

- [0] https://github.com/eeue56/neuro-lingo

Re: Language models as compilers: Simulating pseudocode execution

#47
post #30

Earlier quoted context omitted.

Even regular compilers need quite a bit of nudging to give deterministic results.

Correct me if I'm wrong here, but I am under the impression they're only non-deterministic in the practical sense (i.e, it produces this output on my machine, I can't know what minute differences there are on your machine), but that's not non-deterministic in the truest sense. If you have completely identical inputs you will get the exact same output, ergo, deterministic.

[deleted]

Re: Language models as compilers: Simulating pseudocode execution

#48
Reading the paper, the connection to compilers is more of an analogy rather than a direct technical link.

The authors propose using an LLM to reframe the task as high level psuedocode, and then reason on that code on the specific details of the task

No compilers were used or compiled - no real code was generated or executed. Its just the idea that a programming language syntax has good structure to process details, and a way to interpret some of the results. Many of the other comments here seem like they didn't read the paper at all and are reacting to the headline

Re: Language models as compilers: Simulating pseudocode execution

#49
post #45

Earlier quoted context omitted.

It's better to have a non-deterministic compiler for a task that would be really hard to write an algorithm for otherwise.

But are LLMs really better at algorithm writing than you? I’ve found that they work best when I’ve already pseudocoded the algorithm.

I didn't mean making it write the actual code and using that, but there are tasks that are more error prone or near impossible to write with a traditional approach. So using some zero shot prompting is better than running code.

That's why I find non-determinism as acceptable when otherwise it would be a pain to do something similar.

Re: Language models as compilers: Simulating pseudocode execution

#50
post #10

If you train a LLM to compile, you probably also want to set the randomness to zero, if that is the case you’ve just “brute forced” an actual compiler

you don't want a creative compiler?

Why vote down a rhetorical question?
Post reply on HN