141–150 of 210 posts
Earlier quoted context omitted.
I think MPS stores projects as serialized ASTs and can do VCS merging.
Great. But MPS is not a revision control system.
At this point, I am starting to feel like we don’t need new languages, but new ways to create specifications. I have a hypothesis that an LLM can act as a pseudocode to code translator, where the pseudocode can tolerate a mixture of code-like and natural language specification. The benefit being that it formalizes the human as the specifier (which must be done anyway) and the llm as the code writer. This also might e…
And then we can look at multiple LLM-generated implementations to inform how the prompt might need to be updated further until it's a one-shot.
Now you have perfect intention behind code, and you can refine the intention if it's wrong.
Earlier quoted context omitted.
I think this confuses two different things: - LLMs can act as pseudocode to code translators (they are excellent at this) - LLMs still create bugs and make errors, and a reasonable hypothesis is at a rate in direct proportion to the "complexity" or "buggedness" of the underlying language. In other words, give an AI a footgun and it will happily use it unawares. That doesn't mean however it can't rapidly turn your pse…
Say you have this new language, with only a tiny amount of examples of there. How do the SOTA labs train on you're language? With sufficient examples, it can generate code which gets compiled and then run and that gets fed into a feedback loop to improve upon, but how do you get there? How do you bootstrap that ? Nevermind the dollar cost, how does it offer something above having an LLM generate code in python or Jav…
Languages don't exist in isolation, they exist on a continuum. Your brand new language isn't brand new, it's built off the semantics and syntax of many languages that have come before it. Most language designers operate under what is known as a "weirdness budget", which is about keeping your language to within some delta of other languages modulo a small number of new concepts. This is to maintain comprehensibility, otherwise you get projects like Hoon / Nock where true is false and up is down that no one can figure out.
Under a small weirdness budget, an LLM should be able to understand your new language despite not being trained on it. if you just explain what's different about it. I've had great success with this so far even on early LLM models. One thing you can do is give it the EBNF grammar and it can just generate strings from that. But that method is prone to hallucinations.
Just scanning through this, looks interesting and is totally needed, but I think it is missing showing future use-cases and discussions of decoding. So, for instance, it is all well and good to define a simple language focused on testing and the like, but what about live LLM control and interaction via a programming language? Sort of a conversation in code? Data streams in and function calls stream out with syntax de…
What you want is something that is safe, performant, uses minimal tokens and takes careful note of effects and capabilities. Tests aren't really even important for that use case.
Developed by Jordan Hubbard of NVIDIA (and FreeBSD). My understanding/experience is that LLM performance in a language scales with how well the language is represented in the training data. From that assumption, we might expect LLMs to actually do better with an existing language for which more training code is available, even if that language is more complex and seems like it should be “harder” to understand.
I don’t think that assumption holds. For example, only recently have agents started getting Rust code right on the first try, but that hasn’t mattered in the past because the rust compiler and linters give such good feedback that it immediately fixes whatever goof it made. This does fill up context a little faster, (1) not as much as debugging the problem would have in a dynamic language, and (2) better agentic frame…
I still experience agents slipping in a `todo!` and other hacks to get code to compile, lint, and pass tests.
The loop with tests and doc tests are really nice, agreed, but it'll still shit out bad code.
I can't seem to find in the repo what optimizes this language for LLMs and benchmarks for why it works?
The author's answers are toward the bottom of the README, https://github.com/jordanhubbard/nanolang?tab=readme-ov-file...
Earlier quoted context omitted.
It seems kind of silly that you can’t teach an LLM new tricks though, doesn’t it? This doesn’t sound like an intrinsic limitation and more an artifact of how we produce model weights today.
getting tricks embedded into the weights is expensive, it doesn't happen in a single pass they's why we teach them new tricks on the fly (in-context learning) with instruction files
At this point, I am starting to feel like we don’t need new languages, but new ways to create specifications. I have a hypothesis that an LLM can act as a pseudocode to code translator, where the pseudocode can tolerate a mixture of code-like and natural language specification. The benefit being that it formalizes the human as the specifier (which must be done anyway) and the llm as the code writer. This also might e…
We're already at a point where I think a PR should start with an LLM prompt that fully specs out the change/feature. And then we can look at multiple LLM-generated implementations to inform how the prompt might need to be updated further until it's a one-shot. Now you have perfect intention behind code, and you can refine the intention if it's wrong.