The correct way to do this is with finite model theory but we're not there yet.
Type-constrained code generation with language models
41–50 of 134 posts
Re: Type-constrained code generation with language models
#42Earlier quoted context omitted.
cargo check is WAY faster than go build
Working with both I can say that this is a big no, go mod is as fast if not faster, usually Go dep are much faster because Go does not import as much dependencies as Rust.
In Rust you only need to compile your dependencies once. After that it's just your app because dependencies don't change.
Re: Type-constrained code generation with language models
#43Earlier quoted context omitted.
LLMs work well with any static analysis tool. I frequently instruct Claude to use stuff like “go vet” and “deadcode” when it goes on a tear and writes a bunch of broken trash and declares mission accomplished.
> LLMs work well with any static analysis tool. tsc error messages are so bad that every time my LLM sees one of those "SomeType is not assignable to SomeLongAssTypeDontEvenTryToUnderstandWhatsGoingOnHere >>>>>>>>>>>>>>>>>>>" it just gives up and casts to any. goes for python too.
ha, that's always been my biggest gripe with ts
Re: Type-constrained code generation with language models
#44Earlier quoted context omitted.
This is about the speed with which the compiler can advise an LLM that a particular thing checks or doesn't check. Typescript is much slower than Go
okay so basically the faster compiling means a tigher feedback loop for the LLM to -know- if the code compiles or not etc? interesting is go faster than rust?
> is go faster than rust
No.
They rewrote in go because go is similar enough to typescript, while being faster than typescript.
Source: https://github.com/microsoft/typescript-go/discussions/411
Re: Type-constrained code generation with language models
#45Really cool results! That this research comes out of universities, and not large AI labs, makes me think those labs believe that larger models are still the way to go.
+1 this seems like healthy development
Re: Type-constrained code generation with language models
#46Also worth checking out MultiLSPy, effectively a python wrapper around multiple LSPs: https://github.com/microsoft/multilspy
Used in multiple similar publications, including "Guiding Language Models of Code with Global Context using Monitors" (https://arxiv.org/abs/2306.10763), which uses static analysis beyond the type system to filter out e.g. invalid variable names, invalid control flow etc.
Re: Type-constrained code generation with language models
#47I think TypeScript is uniquely positioned to be the optimal language for LLMs. Tons of training data (benefiting from all the JS examples as well) plus the structure of types for LLMs to follow and tools to enforce.
I can’t be the only one who hopes this was a joke.
Re: Type-constrained code generation with language models
#48Earlier quoted context omitted.
This is about the speed with which the compiler can advise an LLM that a particular thing checks or doesn't check. Typescript is much slower than Go
okay so basically the faster compiling means a tigher feedback loop for the LLM to -know- if the code compiles or not etc? interesting is go faster than rust?
[flagged]
Re: Type-constrained code generation with language models
#49nice. the speed of AI development is accelerating so fast
Re: Type-constrained code generation with language models
#50We (.txt, the outlines people) had a brief thread about this paper on twitter if you're interested: https://x.com/dottxtai/status/1922322194379551128