Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type-constrained code generation with language models

arxiv.org

Post reply on HN
Print view View original

121–130 of 134 posts

Re: Type-constrained code generation with language models

#121

Post by JohnMakin »

_w1tm wrote: post #119
JohnMakin wrote: post #115

Earlier quoted context omitted.

General purpose LLM's fail really hard at this in domains like terraform. There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. Even worse, and probably what exacerbates it, this version matrix changes at a much faster pace than most progra…

> There may be drastic differences in syntax and semantics between the massive matrix of terraform version + provider version(s) and they've shown to be absolutely terrible at navigating that, even if you specify versions specifically. To be fair, humans have trouble with that as well.

That is always the counter but that is not the promise of these tools to be as bad as humans are. Humans can also work their way through it. The context ruts these tools get into with terraform specifically they can never dig out of, it's pretty much worthless for it, at least as many times as I've tried. You will waste far more time trying to figure out where it messed up on something dead simple than if you just looked over a colleague's shoulder, who probably wouldn't run into the same kinds of basic mistakes like making up fields.

Re: Type-constrained code generation with language models

#122

Post by LostBenjamin »

yewW0tm8 wrote: post #120
LostBenjamin wrote: post #114

As an author of this paper, I am very excited see the great discussion here! Several people mentioned the generation - compilation - fixing loop. Just want to remind you that our approach works for not only the generation step but also the fixing step. This is because fixing is essentially asking LLMs to generate a new version of the code. The paper actually has a "repair" experiment to demonstrate this and our appro…

37% gain relative to what? What percent of generated functions were incorrect?

compared to vanilla LLM decoding.

Re: Type-constrained code generation with language models

#123

Post by tpm »

agos wrote: post #113
tpm wrote: post #105

Earlier quoted context omitted.

Can be configured, but then you get to work at a real codebase halfheartedly half-converted from javascript with half the files beginning with ts-ignore. However crappy your Java codebase is going to be, it will still use types. And as just today Gemini hallucinated an API call that never existed (in a widely available and used library even), it's just better to have the ability to check that right away.

the codebases I've worked at in the last ten years are not as half arsed as that - and of course from my point of view are "real" enough. If a codebase is so unkempt the issue is not Typescript - and forgive for writing such a platitude, but you can write awful code in Java, too.

Yes but it will be typed awful code and the typing provides a grounding of sorts. However awful code in untyped/dynamically typed langs can be unspeakably bad. I have many years of Perl experience...

Re: Type-constrained code generation with language models

#125

Post by OutOfHere »

oivey wrote: post #92
kreetx wrote: post #81

Earlier quoted context omitted.

With Haskell, you can be more precise in expressing what you want.

That wouldn’t make a program written in it “more” correct.

It is more correct in a statistical sense over many programs.

Think back to Javascript and untyped Python (without type annotations). It is a lot easier to have bugs in these languages without types. Types help eliminate classes of bugs.

Re: Type-constrained code generation with language models

#126

Post by informal007 »

imtringued wrote: post #90
informal007 wrote: post #71

Earlier quoted context omitted.

Yes, RL works well in fields where answer can be verified in different degree. That's why AlphaGo success, it also should work in code generation and math.

Your reward function can simply be the distance between the constrained output and the unconstrained output, that way you won't even need synthetic data, just a dataset of prompts to RL against.

How to get "unconstrained output" and evaluate the distance between them?

Evaluation method which can decide distance between two sentences is hard to find, best option is closed-source LLM API even it's not the most ideal option. As a result, we also must use current LLM to improve our models.

Re: Type-constrained code generation with language models

#127

Post by int19h »

android521 wrote: post #61
int19h wrote: post #59

Been using Devin for a few months now, for Typescript and Python. I've never seen it check-in uncompilable code, but watching the Devin console I can see it building and using the code to ensure commits are not complete garbage. When it has checked-in compilable and almost right but slightly wrong code, automatically running lint and tests (it doesn't always run them before checking in) from ci triggers it to push a…

is Devin worth the money? Would it be a big jump in productivity migrating from cursor to Devin?

it has been worth it for me, ymmv of course.

also they have a pay-as-you-go tier now as well.

I pay the full $500 though. This month I'm going to blow past the base allowance and tap into 'gift credits'

speaking of which if anyone wants a referral code (gift creds for me, and for you) hmu

Re: Type-constrained code generation with language models

#128

Post by vessenes »

OutOfHere wrote: post #102
vessenes wrote: post #69

Earlier quoted context omitted.

my experience - yes! but. It's more of an edit - compile - fix loop than a write (seemingly) correct code on the first try. This might be a feature. There is a little occasional difficulty on syntax with rust, but there are often the same sort of logic errors / getting lost an llm would have on another codebase -- the compiler helps catch many of these.

Are you saying that the non-looping generators are grossly insufficient for Rust? This matters because the non-looping generators have a fixed monthly subscription cost, whereas the looping ones could cost per call.

I’m not sure what you mean here. All of my coding is done in a loop; something writes, something compiles, something fixes, once compiling, something tests and writes more, repeat until you need bed.

If you mean “can GitHub copilot author long syntactically, type, and memory-safe- correct rust code in one shot?” Then the answer is “not right now”

Re: Type-constrained code generation with language models

#129

Post by vessenes »

koakuma-chan wrote: post #88
vessenes wrote: post #68

Earlier quoted context omitted.

Karpathy nerdbaited me on this last week! I'm almost done with aidocs and aidd. The aidocs server: keeps track of generated llm-friendly docs for any github repo. The aidocs daemon (aidd) is resident, and can watch a repo, find imports in a number of languages, request the docs from aidocs, serve them up in mcp, and/or put them into a directory in your repo. Planning on generating docs for a codebase and incremental…

Hey, I'm definitely interested, but I'm not on MacOS. Please let me know though if there is a GitHub repo I can star or perhaps even send a PR (e.g. no reason for it to only work on MacOS, right?).

What OS do you want? Linux is an obvious next target and “should be” “easy”. :) I’d be happy to have a Linux tester. Windows would be last I think.

Re: Type-constrained code generation with language models

#130

Post by koakuma-chan »

vessenes wrote: post #129
koakuma-chan wrote: post #88

Earlier quoted context omitted.

Hey, I'm definitely interested, but I'm not on MacOS. Please let me know though if there is a GitHub repo I can star or perhaps even send a PR (e.g. no reason for it to only work on MacOS, right?).

What OS do you want? Linux is an obvious next target and “should be” “easy”. :) I’d be happy to have a Linux tester. Windows would be last I think.

I use WSL so yeah, primarily Linux, but I would also be willing to help with Windows if necessary.
Post reply on HN