Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type-constrained code generation with language models

arxiv.org

Post reply on HN
Print view View original

61–70 of 134 posts

Re: Type-constrained code generation with language models

#61

Post by android521 »

int19h wrote: post #59

Been using Devin for a few months now, for Typescript and Python. I've never seen it check-in uncompilable code, but watching the Devin console I can see it building and using the code to ensure commits are not complete garbage. When it has checked-in compilable and almost right but slightly wrong code, automatically running lint and tests (it doesn't always run them before checking in) from ci triggers it to push a…

is Devin worth the money? Would it be a big jump in productivity migrating from cursor to Devin?

Re: Type-constrained code generation with language models

#62

Post by nielstron »

informal007 wrote: post #60

Would it better if we move the feedback loops into RL-stage of LLM training? Are there some related works?

we were thinking about doing exactly this, the closest current work is probably the amazing "Learning Formal Mathematics from Intrinsic Motivation" by Poesia et al (they use constraints too increase the likelihood of generating correct theorems/proofs during RL)

https://arxiv.org/abs/2407.00695

Re: Type-constrained code generation with language models

#64

Post by nielstron »

notnullorvoid wrote: post #14

The general idea seems very promising, I had been hoping someone would do something like this since seeing JSON schema structured outputs for LLMs. Need to dig in a bit more on the implementation, but I was surprised that the paper didn't mention hooking into existing language service/server. There's more than types that an LLM could leverage from existing language tooling. Auto imports is a good example, it is handy…

the problem with LSPs is that they don't guarantee generating a type annotation that we can use for constraints, i.e. we can not ensure the prefix property using LSPs. so we had to roll our own :)

Pulling in more features to help the system is definitely worth looking into!

Re: Type-constrained code generation with language models

#65

Post by nielstron »

jiggawatts wrote: post #2

This was an obvious next step. Most current products can only restrict the token prediction to valid JSON or a specific JSON schema at best. There's no reason that this should be the only grammar available for constrained output mode. The real challenge will be to make this detect and switch languages automatically. For example, a snippet of code could include a LaTeX formula in a comment and SQL in a string literal.…

re detecting and switching language: you could run several constraint systems in parallel and switch as soon as one of them rejects the input and another accepts it

re backtracking: a core part of this paper is ensuring a prefix property. that is there is always a legitimate completion and the model can not "corner" itself!

research needs to be done for what kind of languages and language features this prefix property can be ensured

Re: Type-constrained code generation with language models

#67

Post by primitivesuave »

ArcaneMoose wrote: post #8

I think TypeScript is uniquely positioned to be the optimal language for LLMs. Tons of training data (benefiting from all the JS examples as well) plus the structure of types for LLMs to follow and tools to enforce.

Completely agree. Even with the basic LLMs in the $20/month Cursor plan, I can work 10x faster on TypeScript codebases than I could otherwise, while for Python that multiple feels more like 2-3x. The autocompletions are especially impressive when there is a well-organized type system.

Also in response to adjacent commenters - many mission-critical TS codebases will disable the use of an explicit "any" with eslint - https://typescript-eslint.io/rules/no-explicit-any/.

Re: Type-constrained code generation with language models

#68

Post by vessenes »

koakuma-chan wrote: post #29

The vibe code society would benefit way more if libraries hosted their docs in a way that's easy to copy and paste into an LLM.

Karpathy nerdbaited me on this last week! I'm almost done with aidocs and aidd.

The aidocs server: keeps track of generated llm-friendly docs for any github repo.

The aidocs daemon (aidd) is resident, and can watch a repo, find imports in a number of languages, request the docs from aidocs, serve them up in mcp, and/or put them into a directory in your repo. Planning on generating docs for a codebase and incremental docs creation later.

I could use a couple beta testers -- lmk if you're interested. macos for now, although the daemon is written in go and should be portable.

Re: Type-constrained code generation with language models

#69

Post by vessenes »

koakuma-chan wrote: post #33
mindwok wrote: post #30

Earlier quoted context omitted.

Yep, and Rust famously goes beyond this by modelling memory ownership at compile time. In fact, the more behaviour we can model at compile time the better when it comes to LLMs - there's some cool ideas here like transpiling Rust into languages for formal verification. See https://github.com/formal-land/coq-of-rust as an example. Formal verification was one of those things that was previously so annoying to do that i…

Can LLMs properly code in Rust yet? There is way more TypeScript code out there compared to Rust, and I doubt structured output can alleviate this.

my experience - yes! but. It's more of an edit - compile - fix loop than a write (seemingly) correct code on the first try. This might be a feature.

There is a little occasional difficulty on syntax with rust, but there are often the same sort of logic errors / getting lost an llm would have on another codebase -- the compiler helps catch many of these.

Re: Type-constrained code generation with language models

#70

Post by NitpickLawyer »

vessenes wrote: post #69
koakuma-chan wrote: post #33

Earlier quoted context omitted.

Can LLMs properly code in Rust yet? There is way more TypeScript code out there compared to Rust, and I doubt structured output can alleviate this.

my experience - yes! but. It's more of an edit - compile - fix loop than a write (seemingly) correct code on the first try. This might be a feature. There is a little occasional difficulty on syntax with rust, but there are often the same sort of logic errors / getting lost an llm would have on another codebase -- the compiler helps catch many of these.

> This might be a feature.

I think so as well. The rust errors are some of the most "helpful" and easy to understand (once you grok the core concepts around rust) and it seems that the loop of - generate (maybe constrained) - check - fix benefits from this. In my testing it is better than python (long ass traces that you have to manually trim for the LLM).

Post reply on HN