91–100 of 105 posts
> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…
I do applaud the lang design effort as there are countless routes of accepting Jesus as your savior.
This is exactly the wrong approach. LLMs are good at writing programming languages they already know, that are well represented in the training data, not at writing programming languages that they have never seen before, so that you have to include the entire programming language manual and lots of example code in every prompt.
This is not my experience. I've been experimenting with something very similar to vera. However my language transpiles into multiple languages (Java, Typescript, Common Lisp, Rust, C++, Python, C# and Swift). The transpiler is written in the language itself (there's a separate bootstrap transpiler written in Common Lisp). But where I'm going is that Claude, at least, is extremely capable at writing decent code in my…
LLMs are stateless, they don't "remember" your bespoke programming language manual and examples between completion calls, so you have to repeatedly include all that with each and every completion call, which balloons the number of tokens used, reduces how much useful work you can do with the remaining tokens and attention, and is a costly waste of tokens and electricity and money.
That isn't anywhere near as effective or efficient as using the LLM's pre-existing training on billions of lines of well known programming languages, manuals, tutorials, examples, code bases, stack overflow discussions, books, github repos, pr's, etc.
What is your extraordinary evidence for your extraordinary claims? Have you empirically measured how well it works, or is it just vibes and handwaving?
> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…
> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…
My Spidey sense was tingling when I saw that, too. An additional issue is how humans are supposed to read the code at all so that they can provide help to the LLM if it’s off track. If the code is only usable by models, the models need to be good enough to deal with binary feedback (“Code doesn’t work.”). The human won’t be able to read the code and steer the model. Given the levels of steering required today, that m…
There are many problems we will need to address in the future. A programming language that is easy for machines to write but hard for humans to read isn’t one of them.
This reminds me of Dafny: https://dafny.org/
Actually, that's an interesting question: how good are LLMs at writing Dafny?
> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…
Seems like a weird decision taken from a weird paper that make everything harder for humans AND llms. Variables names give useful context when correctly named.