Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type-constrained code generation with language models

arxiv.org

Post reply on HN
Print view View original

71–80 of 134 posts

Re: Type-constrained code generation with language models

#71

Post by informal007 »

nielstron wrote: post #62
informal007 wrote: post #60

Would it better if we move the feedback loops into RL-stage of LLM training? Are there some related works?

we were thinking about doing exactly this, the closest current work is probably the amazing "Learning Formal Mathematics from Intrinsic Motivation" by Poesia et al (they use constraints too increase the likelihood of generating correct theorems/proofs during RL) https://arxiv.org/abs/2407.00695

Yes, RL works well in fields where answer can be verified in different degree. That's why AlphaGo success, it also should work in code generation and math.

Re: Type-constrained code generation with language models

#73

Post by agos »

treyd wrote: post #57
marviel wrote: post #24

Earlier quoted context omitted.

what do you dislike about it?

TypeScript is arguably one of the weaker statically typed languages, with how it allows `any` to quietly violate the type checked assumptions. It makes it harder to do a lot of the basic typing mistakes in JS, but it doesn't prevent them by any means, especially if you have to interface with (typeless) JS code. So for these reasons alone I would be against using TS as a lingua franca for LLM codegen (as is GP I assum…

tsc can be configured to avoid implicit use of any ("noImplicitAny": true) and ESLint can be set up to avoid explicit use of any. Typeless JS code is also a thing of the past.

But the devil is in the details - some libraries are typed quite crappily, some have unnecessary complex types, and the code that the LLMs was trained on is probably not the best in the world

Re: Type-constrained code generation with language models

#74

Post by johnmw »

ArcaneMoose wrote: post #8

I think TypeScript is uniquely positioned to be the optimal language for LLMs. Tons of training data (benefiting from all the JS examples as well) plus the structure of types for LLMs to follow and tools to enforce.

Those who agree might be interested in "Introducing TypeChat" by Anders Hejlsberg + others (2023) [1]

[1]: https://microsoft.github.io/TypeChat/blog/introducing-typech...

Re: Type-constrained code generation with language models

#75

Post by Corey_ »

I completely agree that TypeScript is ideal for LLMs. The type system and the extensive training data make it the best choice. But as someone who's been working with TypeScript for a while, I still see LLMs struggling with complex generics or even simple types. It’s better than before, but still far from perfect.

Also, TypeScript error messages can be a pain. When LLMs encounter something like "SomeType is not assignable," instead of handling it properly, they often just cast it to any. This happens way too often.

Re: Type-constrained code generation with language models

#76

Post by dcsan »

johnmw wrote: post #74
ArcaneMoose wrote: post #8

I think TypeScript is uniquely positioned to be the optimal language for LLMs. Tons of training data (benefiting from all the JS examples as well) plus the structure of types for LLMs to follow and tools to enforce.

Those who agree might be interested in "Introducing TypeChat" by Anders Hejlsberg + others (2023) [1] [1]: https://microsoft.github.io/TypeChat/blog/introducing-typech...

Wish this project had more traction. Typechat with type checking could generate lots of synthetic data for model training too

Re: Type-constrained code generation with language models

#77

Post by nielstron »

_jayhack_ wrote: post #46

Also worth checking out MultiLSPy, effectively a python wrapper around multiple LSPs: https://github.com/microsoft/multilspy Used in multiple similar publications, including "Guiding Language Models of Code with Global Context using Monitors" ( https://arxiv.org/abs/2306.10763 ), which uses static analysis beyond the type system to filter out e.g. invalid variable names, invalid control flow etc.

Yes this work is super cool too! Note that LSPs can not guarantee resolving the necessary types that we use to ensure the prefix property, which we leverage to avoid backtracking and generation loops.

Re: Type-constrained code generation with language models

#78

Post by oivey »

OutOfHere wrote: post #22
ArcaneMoose wrote: post #8

I think TypeScript is uniquely positioned to be the optimal language for LLMs. Tons of training data (benefiting from all the JS examples as well) plus the structure of types for LLMs to follow and tools to enforce.

There are languages that constrain types a lot more tightly than TypeScript, e.g. Kotlin, Rust, and Haskell. The more constrained the types, the more correct the program could be.

The program won’t be “more” correct. What would that even mean? Writing correct programs might be easier (or not) with more “constrained” (ill defined) typing.

Re: Type-constrained code generation with language models

#79

Post by energy123 »

This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it.

Google somewhat did this with javascript in their latest Gemini-2.5 Pro release. But what about doing it for a smaller language? Google isn't going to do that, but there is still a lot of demand.

Post reply on HN