Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type-constrained code generation with language models

arxiv.org

Post reply on HN
Print view View original

101–110 of 134 posts

Re: Type-constrained code generation with language models

#101

Post by tough »

eigenspace wrote: post #100
tough wrote: post #99

Earlier quoted context omitted.

it feels to me most of the real usage of AI is in coding right now, so a small lab that decided to go all in into just code-gen would have at least the differentiator of a narrower field to beat the bigger incumbents doing it all? I dunno tho. Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ? Once you're into real usage territory, you can't n…

Again though, my point was just that it's not actually clear that you can do better than these big models by taking a narrower focus. I'm saying that that the things these big LLMs are learning about other languages probably do have utility when applied even to quite niche languages. If you take some niche language and build an LLM from scratch that's hyperspecialized on that language, will that LLM actually outperfo…

So how I envision this would be like a dual system, you let the frontier bigger LLM come up with the overall function signature, structure, and reasoning/planning around the specific code, but then have it ask the hyperspecialized fine-tuned model which can only output valid code, to create it.

You get then best of both worlds at the expense of a double-round trip or x2, which for something like coding seems fine, people are OK paying 200 for ChatGPT Plus

This also would solve the context window sizes problem of them getting full and the model starting to generate non-sense, if you have the bigger model using the bigger context window to orchestrate and organize the task calling smaller specialized sub-modules, that seems like it should yield better final code outptus than just one big ass LLM

but we'r'e moving the goalposts from 1 model to multi-agentic system i guess so nvermind

and i agree it seems all the big corps are betting for bigger more data for now

Re: Type-constrained code generation with language models

#102

Post by OutOfHere »

vessenes wrote: post #69
koakuma-chan wrote: post #33

Earlier quoted context omitted.

Can LLMs properly code in Rust yet? There is way more TypeScript code out there compared to Rust, and I doubt structured output can alleviate this.

my experience - yes! but. It's more of an edit - compile - fix loop than a write (seemingly) correct code on the first try. This might be a feature. There is a little occasional difficulty on syntax with rust, but there are often the same sort of logic errors / getting lost an llm would have on another codebase -- the compiler helps catch many of these.

Are you saying that the non-looping generators are grossly insufficient for Rust? This matters because the non-looping generators have a fixed monthly subscription cost, whereas the looping ones could cost per call.

Re: Type-constrained code generation with language models

#103

Post by harperlee »

eigenspace wrote: post #98
energy123 wrote: post #79

This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…

I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.

From my experience around less-used languages (with clojure on one hand and code aster's python on the other), LLMs may be able to generalize syntax but availability of APIs, functions, etc. is something that you can't solve by generalizing. Or more precisely, you can generalize but that means hallucinating non existing tools.

Re: Type-constrained code generation with language models

#104

Post by unshavedyak »

harperlee wrote: post #103
eigenspace wrote: post #98

Earlier quoted context omitted.

I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.

From my experience around less-used languages (with clojure on one hand and code aster's python on the other), LLMs may be able to generalize syntax but availability of APIs, functions, etc. is something that you can't solve by generalizing. Or more precisely, you can generalize but that means hallucinating non existing tools.

Would non-generalizing solve this issue for libraries though? Ie a lot of models produce reasonable code for me, but i almost always care about usage of libraries. That's where they get the wrong version, or hallucinate, or etc for me.

Re: Type-constrained code generation with language models

#105

Post by tpm »

agos wrote: post #73
treyd wrote: post #57

Earlier quoted context omitted.

TypeScript is arguably one of the weaker statically typed languages, with how it allows `any` to quietly violate the type checked assumptions. It makes it harder to do a lot of the basic typing mistakes in JS, but it doesn't prevent them by any means, especially if you have to interface with (typeless) JS code. So for these reasons alone I would be against using TS as a lingua franca for LLM codegen (as is GP I assum…

tsc can be configured to avoid implicit use of any ("noImplicitAny": true) and ESLint can be set up to avoid explicit use of any. Typeless JS code is also a thing of the past. But the devil is in the details - some libraries are typed quite crappily, some have unnecessary complex types, and the code that the LLMs was trained on is probably not the best in the world

Can be configured, but then you get to work at a real codebase halfheartedly half-converted from javascript with half the files beginning with ts-ignore.

However crappy your Java codebase is going to be, it will still use types. And as just today Gemini hallucinated an API call that never existed (in a widely available and used library even), it's just better to have the ability to check that right away.

Re: Type-constrained code generation with language models

#106

Post by nurettin »

energy123 wrote: post #79

This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…

Using the language itself isn't the challenge for LLMs, they do that with a very high success rate. I haven't seen an LLM make syntax errors for several months. Calling the right functions with correct parameters is the challenge your hypothetical AI lab will have to solve (or half ass it and show great benchmark results).

Re: Type-constrained code generation with language models

#107

Post by nulld3v »

koakuma-chan wrote: post #42
Thaxll wrote: post #34

Earlier quoted context omitted.

Working with both I can say that this is a big no, go mod is as fast if not faster, usually Go dep are much faster because Go does not import as much dependencies as Rust.

In Rust you only need to compile your dependencies once. After that it's just your app because dependencies don't change.

This may be true, but in my experience Rust is still slower to compile because monomorphization must be done for deps every time you compile, even for deps that are already compiled. And monomorphization ends up taking a long time because it is done on every type/function that uses generics, and Rust code tends to use generics very liberally.

Re: Type-constrained code generation with language models

#108

Post by _alternator_ »

Until a couple weeks ago, I considered this a promising approach. What changed? Agents, and Claude code in particular.

My prior experience was that LLMs were not much better than reading the docs, and certainly you wouldn’t get far vibe-coding in Rust. But Claude code behaves like I would, writing code that does t compile (typical LLM behavior), then reading the errors, correcting the code, and iterating until it compiles.

Its first attempt at a graph based scheduler in Rust took about $3 and 10 minutes to work correctly. It was ~500 loc, so definitely faster than what I can write in rust. (To be fair I spent a decent amount of time drafting a description of what I wanted in a markdown file to get Claude started).

Re: Type-constrained code generation with language models

#109

Post by gdiamos »

_alternator_ wrote: post #108

Until a couple weeks ago, I considered this a promising approach. What changed? Agents, and Claude code in particular. My prior experience was that LLMs were not much better than reading the docs, and certainly you wouldn’t get far vibe-coding in Rust. But Claude code behaves like I would, writing code that does t compile (typical LLM behavior), then reading the errors, correcting the code, and iterating until it com…

If you have two methods that both improve accuracy, why not stack them?
Post reply on HN