Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vera: a programming language designed for machines to write

github.com

Post reply on HN
Print view View original

11–20 of 105 posts

Re: Vera: a programming language designed for machines to write

#11

Post by sornaensis »

danpalmer wrote: post #5

> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…

I'm curious what issues you had with haskell? I have had the opposite experience and find them dreadful at Java et al.

Surely, denser languages should be better for LLMs?

Re: Vera: a programming language designed for machines to write

#13

Post by DonHopkins »

This is exactly the wrong approach. LLMs are good at writing programming languages they already know, that are well represented in the training data, not at writing programming languages that they have never seen before, so that you have to include the entire programming language manual and lots of example code in every prompt.

Re: Vera: a programming language designed for machines to write

#14

Post by danpalmer »

sornaensis wrote: post #11
danpalmer wrote: post #5

> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…

I'm curious what issues you had with haskell? I have had the opposite experience and find them dreadful at Java et al. Surely, denser languages should be better for LLMs?

Density is a double edged sword. On the one hand you want to minimise context usage, but on the other hand more text on the page means more that the LLM can work with.

Re: Vera: a programming language designed for machines to write

#15

Post by sas41 »

I find the claims regarding LLMs and their mistake prone nature around variable names very confusing.

It appears that me and creator have had vastly different experiences with LLMs and their capabilities with complex code bases and complicated business logic.

My observations point to LLMs being much more successful when variables and methods have explicit, detailed names, it's the best way to keep them on track and minimize the chance of confusion, next closest thing being explicit comments and inline documentation.

Poorly named and poorly documented things in a codebase only cause it to reason more on what it could be, often reaching a (wrong) conclusion, wasting tokens, wasting time.

Perhaps this diversion in philosophy is due to fundamental differences in how we view the tool at hand.

I do not trust the machine, as such I review it's output, and if the variables lacked names, that would be significantly harder. But if I had a "Jesus, take the wheel!" attitude, perhaps I'd care far less.

Re: Vera: a programming language designed for machines to write

#16

Post by svachalek »

rtpg wrote: post #12

The lack of naming seems to indicate a fundamental misunderstanding of how LLM coding agents are successful, and just makes me doubt anything about this project being useful and workable.

Yeah it seems based on 2023 research which is ancient, back when we didn't have coding agents at all, and on some 1980s sci fi concepts of "how machines think" (beedeeboop) rather than the all too human coding agents we have.

If I had to design one of these, I'd go for:

1. Token minimization (which may be circular, I'm sure tokens are selected for these models at least in part based on syntax of popular languages)

2. As many compile time checks as possible (good for humans, even better for machines with limited context)

3. Maximum locality. That is, a feature can largely be written in one file, rather than bits and pieces all over the codebase. Because of how context and attention work. This is the one I don't see much in commercially popular languages. It's more of a declarative thing, "configuration driven development".

Re: Vera: a programming language designed for machines to write

#17

Post by robviren »

danpalmer wrote: post #5

> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…

I too have found the models do well with Go. I will say despite the backwards compatibility guarantee library API changes, what counts as "good" patterns, and new language additions do add some friction to the experience. Almost always works but it can be a bit inconsistent in how the code shows up.

Re: Vera: a programming language designed for machines to write

#19

Post by Octoth0rpe »

svachalek wrote: post #16
rtpg wrote: post #12

The lack of naming seems to indicate a fundamental misunderstanding of how LLM coding agents are successful, and just makes me doubt anything about this project being useful and workable.

Yeah it seems based on 2023 research which is ancient, back when we didn't have coding agents at all, and on some 1980s sci fi concepts of "how machines think" (beedeeboop) rather than the all too human coding agents we have. If I had to design one of these, I'd go for: 1. Token minimization (which may be circular, I'm sure tokens are selected for these models at least in part based on syntax of popular languages) 2.…

> That is, a feature can largely be written in one file, rather than bits and pieces all over the codebase.

This seems to be at odds with the goal of token minimization. Lots of small files that are narrowly scoped means less has to be loaded into context when making a change, right?

Throwing out another idea: I wonder if we could see some kind of equivalent of c header files for more modern languages so that an llm just has to read the equivalent of a .h file to start using a library.

Re: Vera: a programming language designed for machines to write

#20

Post by hgoel »

sornaensis wrote: post #11
danpalmer wrote: post #5

> The empirical literature shows that models are particularly vulnerable to naming-related errors like choosing misleading names, reusing names incorrectly, and losing track of which name refers to which value. I think Vera might be missing something here. In my experience, LLMs code better the less of a mental model you need, vs the more is in text on the page. Go – very little hidden, everything in text on the page…

I'm curious what issues you had with haskell? I have had the opposite experience and find them dreadful at Java et al. Surely, denser languages should be better for LLMs?

The context window also limits how deeply the model can "think", and it does this in natural language. So a language suited to LLMs would have balanced density, if it's too dense, the model spends many tokens working through the logic, if it's too sparse, it spends many tokens to read/write the code.

I think in the context of already trained LLMs, the languages most suited to LLMs are also the ones most suited to humans. Besides just having the most code to train on, humans also face similar limitations, if the language is too dense they have to be very careful in considering how to do something, if it's too sparse, the code becomes a pain to maintain.

Post reply on HN