This is anticipated from work on constrained output from LLMs, and it's good to see it being developed. One nitpick though, this paper mentions the complexities of implementing type checking for program prefixes in languages that are not context free. It's true this is extremely difficult for languages which are context sensitive, especially because types may be defined after they are used. However, it does not mention that it is impossible to implement such a program for Turing complete languages such as C++. I would never miss such an opportunity to criticize C++ and highlight the need for better language design. I love you C++.
Type-constrained code generation with language models
91–100 of 134 posts
Re: Type-constrained code generation with language models
#92Earlier quoted context omitted.
The program won’t be “more” correct. What would that even mean? Writing correct programs might be easier (or not) with more “constrained” (ill defined) typing.
With Haskell, you can be more precise in expressing what you want.
That wouldn’t make a program written in it “more” correct.
Re: Type-constrained code generation with language models
#93They should extend this to Haskell and make use of the Curry-Howard isomorphism: define the program you want by a type signature and have the LLM find the implementation.
The classic approach to this is Djinn https://hackage.haskell.org/package/djinn It's not very good with concrete types (Text, Int, MyCustomThing, etc.) but is good for polymorphic functions whose parts only fit together in a few ways.
More recent work is better at using concrete types, and choosing functions from Hackage, like Hoogle+ https://github.com/TyGuS/hoogle_plus and Hectare https://dl.acm.org/doi/10.1145/3547622
There's also "inductive programming" (producing a function from input/output examples), with Haskell implementations like Magic Haskeller http://nautilus.cs.miyazaki-u.ac.jp/~skata/MagicHaskeller.ht...
Re: Type-constrained code generation with language models
#94This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…
It makes sense to specialize it on one programming language to dedicate all of the LLM's intellectual space to that one domain, but on the flip side I wonder how much the LLM's sharpness and reasoning capabilities is increased by having more data to train on even if it's the wrong programming language.
As a developer I certainly think my programming skills in a specific language was improved by knowing other languages so I can contrast and compare.
Re: Type-constrained code generation with language models
#95Earlier quoted context omitted.
https://ai-sdk.dev/llms.txt
Depends on the library I guess, I spent 12~ hours today vibe coding with LiveKit and their /llms.txt is https://docs.livekit.io/llms.txt
what i do if no good llms txt is to download the whole docs from gh or website and keep the MD files available to my agent via a small mcp server
Re: Type-constrained code generation with language models
#96This is anticipated from work on constrained output from LLMs, and it's good to see it being developed. One nitpick though, this paper mentions the complexities of implementing type checking for program prefixes in languages that are not context free. It's true this is extremely difficult for languages which are context sensitive, especially because types may be defined after they are used. However, it does not menti…
noted. we'll make sure to critizise turing complete type systems more thoroughly next time :))
Re: Type-constrained code generation with language models
#97This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…
It makes sense to specialize it on one programming language to dedicate all of the LLM's intellectual space to that one domain, but on the flip side I wonder how much the LLM's sharpness and reasoning capabilities is increased by having more data to train on even if it's the wrong programming language. As a developer I certainly think my programming skills in a specific language was improved by knowing other language…
You could just have specialized fine-tunes for esxh programling la guage that are only called when writing code, a more general bigger model could pass the plan/pseudo code to it
Re: Type-constrained code generation with language models
#98This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…
I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax.
Many people do think this, but I'm not sure many of them are running AI labs.
Re: Type-constrained code generation with language models
#99This is what I'd consider doing if I was a small AI lab. Don't try to build a frontier LLM that beats all benchmarks. Try to make the world's best LLM at one programming language. Create your RL pipeline that puts all your resources into making the LLM the best at that language. Even better if there's a dearth of human-created training data on Github, since all your competitors will be bad at it. Google somewhat did…
I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.
it feels to me most of the real usage of AI is in coding right now, so a small lab that decided to go all in into just code-gen would have at least the differentiator of a narrower field to beat the bigger incumbents doing it all?
I dunno tho.
Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ?
Once you're into real usage territory, you can't no longer use make up numbers to justify future growth.
Re: Type-constrained code generation with language models
#100Earlier quoted context omitted.
I'm not saying this is a bad idea, but it does sound like a rather risky prospect. You're basically proposing a bet against the ability of LLMs to generalize across programming languages, and to embed concepts at a deeper level than the syntax. Many people do think this, but I'm not sure many of them are running AI labs.
it feels to me most of the real usage of AI is in coding right now, so a small lab that decided to go all in into just code-gen would have at least the differentiator of a narrower field to beat the bigger incumbents doing it all? I dunno tho. Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ? Once you're into real usage territory, you can't n…
Again though, my point was just that it's not actually clear that you can do better than these big models by taking a narrower focus. I'm saying that that the things these big LLMs are learning about other languages probably do have utility when applied even to quite niche languages.
If you take some niche language and build an LLM from scratch that's hyperspecialized on that language, will that LLM actually outperform some big LLM that's trained on all the programming resources out there, and all the blogs, forum conversations, stack overflow posts on all those languages, and then learns to generalize that information and apply it to your niche language?
One of the things that LLMs seem to excel at is taking information from one context, transforming it and applying it to another context.