Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type-constrained code generation with language models

arxiv.org

Post reply on HN
Print view View original

111–120 of 134 posts

Re: Type-constrained code generation with language models

#111

Post by robrenaud »

eigenspace wrote: post #98
energy123 wrote: post #79

This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…

I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.

Meta synthetically generated lots of PHP from Python for Llama 3 for training purposes. Meta writes a crazy amount of PHP internally. Translation tends to be way easier than unconstrained generation for LLMs. But if you can translate and filter a large amount of code, you can learn to generate. If you also translate and run the unittests, you get another layer of error checking.

https://arxiv.org/abs/2407.21783

See figure 8.

Re: Type-constrained code generation with language models

#112

Post by _alternator_ »

gdiamos wrote: post #109
_alternator_ wrote: post #108

Until a couple weeks ago, I considered this a promising approach. What changed? Agents, and Claude code in particular. My prior experience was that LLMs were not much better than reading the docs, and certainly you wouldn’t get far vibe-coding in Rust. But Claude code behaves like I would, writing code that does t compile (typical LLM behavior), then reading the errors, correcting the code, and iterating until it com…

If you have two methods that both improve accuracy, why not stack them?

This reasoning is at odds with recent history. A lot of effort was put into making machines that could only output grammatically correct sentences. Yet LLMs output grammatically correct sentences when needed, and a whole lot more as context demands, without ever hardcoding it. It would be foolish to restrict LLMs to only grammatical sentences at this point, but it might be a good tool to run on generated docs because sometimes words are misspelled or sentences are confusing.

Similarly, we have a tool that makes sure the type and syntax are correct, namely, a compiler. Building an LLM that can only output syntactically correct code is one approach of stacking them, but in fact it will significantly worsen the ability of the LLM to reason and construct code. The winning choice seems to be to emulate the workflow of humans: code, compile, read errors, repeat.

Re: Type-constrained code generation with language models

#113

Post by agos »

tpm wrote: post #105
agos wrote: post #73

Earlier quoted context omitted.

tsc can be configured to avoid implicit use of any ("noImplicitAny": true) and ESLint can be set up to avoid explicit use of any. Typeless JS code is also a thing of the past. But the devil is in the details - some libraries are typed quite crappily, some have unnecessary complex types, and the code that the LLMs was trained on is probably not the best in the world

Can be configured, but then you get to work at a real codebase halfheartedly half-converted from javascript with half the files beginning with ts-ignore. However crappy your Java codebase is going to be, it will still use types. And as just today Gemini hallucinated an API call that never existed (in a widely available and used library even), it's just better to have the ability to check that right away.

the codebases I've worked at in the last ten years are not as half arsed as that - and of course from my point of view are "real" enough.

If a codebase is so unkempt the issue is not Typescript - and forgive for writing such a platitude, but you can write awful code in Java, too.

Re: Type-constrained code generation with language models

#114

Post by LostBenjamin »

As an author of this paper, I am very excited see the great discussion here!

Several people mentioned the generation - compilation - fixing loop. Just want to remind you that our approach works for not only the generation step but also the fixing step. This is because fixing is essentially asking LLMs to generate a new version of the code. The paper actually has a "repair" experiment to demonstrate this and our approach achieves significant gain in this experiment, i.e., 37% relative improvement in functional correctness.

Re: Type-constrained code generation with language models

#115

Post by JohnMakin »

harperlee wrote: post #103
eigenspace wrote: post #98

Earlier quoted context omitted.

I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.

From my experience around less-used languages (with clojure on one hand and code aster's python on the other), LLMs may be able to generalize syntax but availability of APIs, functions, etc. is something that you can't solve by generalizing. Or more precisely, you can generalize but that means hallucinating non existing tools.

General purpose LLM's fail really hard at this in domains like terraform. There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. Even worse, and probably what exacerbates it, this version matrix changes at a much faster pace than most programming languages typically introduce large changes.

Re: Type-constrained code generation with language models

#116

Post by binary132 »

koakuma-chan wrote: post #89
binary132 wrote: post #51

Earlier quoted context omitted.

that is also the case in Go…?

Sure, but the point is: don't be scared of dependencies in Rust.

Well the context was a comparison of pros and cons and you started with “in Rust” so perhaps you can see why it sounded like you were presenting it as a pro.

Re: Type-constrained code generation with language models

#117

Post by thesz » 

  > To address this challenge, we introduce a type-constrained decoding approach that leverages type systems to guide code generation.
This should not work with type inference even at the level of C++ "auto x = " - "auto" does not constrain "x" at all and what is right of equal sign is not constrained either..

In Haskell, the gap is even wider. A long "where" clause may have dependencies constraining things in different direction.

But, what important I see here is the continuation of reinvention of Cyc, from different starting point. ;)

Definitely, "every big LLM has in support code an ad-hoc bug ridden inefficient implementation of half of Cyc." Cyc was written in Lisp, most of LLM support code is C/C++, thus, it is just a corrolary of Greenspun's Tenth Rule.

Re: Type-constrained code generation with language models

#118

Post by tough »

LostBenjamin wrote: post #114

As an author of this paper, I am very excited see the great discussion here! Several people mentioned the generation - compilation - fixing loop. Just want to remind you that our approach works for not only the generation step but also the fixing step. This is because fixing is essentially asking LLMs to generate a new version of the code. The paper actually has a "repair" experiment to demonstrate this and our appro…

Thank you for your research really impressive work!

Re: Type-constrained code generation with language models

#119

Post by _w1tm »

JohnMakin wrote: post #115
harperlee wrote: post #103

Earlier quoted context omitted.

From my experience around less-used languages (with clojure on one hand and code aster's python on the other), LLMs may be able to generalize syntax but availability of APIs, functions, etc. is something that you can't solve by generalizing. Or more precisely, you can generalize but that means hallucinating non existing tools.

General purpose LLM's fail really hard at this in domains like terraform. There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. Even worse, and probably what exacerbates it, this version matrix changes at a much faster pace than most progra…

> There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically.

To be fair, humans have trouble with that as well.

Re: Type-constrained code generation with language models

#120

Post by yewW0tm8 »

LostBenjamin wrote: post #114

As an author of this paper, I am very excited see the great discussion here! Several people mentioned the generation - compilation - fixing loop. Just want to remind you that our approach works for not only the generation step but also the fixing step. This is because fixing is essentially asking LLMs to generate a new version of the code. The paper actually has a "repair" experiment to demonstrate this and our appro…

37% gain relative to what? What percent of generated functions were incorrect?
Post reply on HN