Almost no positive comments. I for one find this a great try, a very interesting project. I hope this kind of experiments gain track.
Nanolang: A tiny experimental language designed to be targeted by coding LLMs
51–60 of 210 posts
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#52Developed by Jordan Hubbard of NVIDIA (and FreeBSD). My understanding/experience is that LLM performance in a language scales with how well the language is represented in the training data. From that assumption, we might expect LLMs to actually do better with an existing language for which more training code is available, even if that language is more complex and seems like it should be “harder” to understand.
I wonder if there is a way to create a sort of 'transpilation' layer to a new language like this for existing languages, so that it would be able to use all of the available training from other languages. Something that's like AST to AST. Though I wonder if it would only work in the initial training or fine-tuning stage.
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#53I feel this could be achieved better with Golang or Kotlin and a custom linter that enforces parentheses around each expression term to make precedence explicit, and enforce each function has at least one test. Although I guess neither of those languages has free interop with C, they are close. And Go doesn’t have unions :’(
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#54A language targeting an LLM might be well served with a lot of keywords, similar to a CISC instruction set, where keywords do specific things well. Giving it building blocks and having them piece together is likely to pay off.
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#55One novel part here is every function is required to have tests that run at compile time. I'm still skeptical of the value add having to teaching a custom language to an LLM instead of using something like lua or python and applying constraints like test requirements onto that.
Pyret, a teaching language for CS, in the vein of Racket, does require testing by writing functions.
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#56At this point, I am starting to feel like we don’t need new languages, but new ways to create specifications. I have a hypothesis that an LLM can act as a pseudocode to code translator, where the pseudocode can tolerate a mixture of code-like and natural language specification. The benefit being that it formalizes the human as the specifier (which must be done anyway) and the llm as the code writer. This also might e…
>>new ways to create specifications. Thats again programming languages. Real issue with LLMs now is it doesn't matter if it can generate code quickly. Some one still has to read, verify and test it. Perhaps we need a need a terse programming language. Which can be read quickly and verified. You could call that specification.
Yes, essentially a higher level programming language than what we currently have. A programming language that doesn't have strict syntax, and can be expressed with words or code. And like any other programming language, it includes specifications for the tests and expectations of the result.
The programming language can look more like code in parts where the specification needs to be very detailed. I think people can get intuition about where the LLM is unlikely to be successful. It can have low detail for boilerplate or code that is simple to describe.
You should be able to alter and recompile the specification, unlike the wandering prompt which makes changes faster than normal version control practices keep up with.
Perhaps there's a world where reading the specification rather than the compiled code is sufficient in order to keep cognitive load at reasonable levels.
At very least, you can read compiled code until you can establish your own validation set and create statistical expectations about your domain. Principally, these models will always be statistical in nature. So we probably need to start operating more inside that kind of framework if we really want to be professional about it.
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#57Author has a Wikipedia page about him linked from his GitHub profile: https://en.wikipedia.org/wiki/Jordan_Hubbard
Summary:
- Co-created FreeBSD.
- Led UNIX technologies at Apple for 13 years
- iXSystems, lead FreeNAS
- idk something about Uber
- Senior Director for GPU Compute Software at NVIDIA
For whatever it’s worth.
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#58At this point, I am starting to feel like we don’t need new languages, but new ways to create specifications. I have a hypothesis that an LLM can act as a pseudocode to code translator, where the pseudocode can tolerate a mixture of code-like and natural language specification. The benefit being that it formalizes the human as the specifier (which must be done anyway) and the llm as the code writer. This also might e…
This is the approach that Agint takes. We inference the structure of the code first top down as a graph, then add in types, then interpret the types as in out function signatures and then "inpaint" the functions for codegen.
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#59At this point, I am starting to feel like we don’t need new languages, but new ways to create specifications. I have a hypothesis that an LLM can act as a pseudocode to code translator, where the pseudocode can tolerate a mixture of code-like and natural language specification. The benefit being that it formalizes the human as the specifier (which must be done anyway) and the llm as the code writer. This also might e…
I’ve been on a similar train of thought. Just last weekend I built a little experiment, using LLMs to highlight pseudocode syntax:
Re: Nanolang: A tiny experimental language designed to be targeted by coding LLMs
#60Developed by Jordan Hubbard of NVIDIA (and FreeBSD). My understanding/experience is that LLM performance in a language scales with how well the language is represented in the training data. From that assumption, we might expect LLMs to actually do better with an existing language for which more training code is available, even if that language is more complex and seems like it should be “harder” to understand.
A lot of this depends on your workflow. A language with great typing, type checking and good compiler errors will work better in a loop than one with a large surface overhead and syntax complexity, even if it's well represented. This is the instinct behind, e.g. https://github.com/toon-format/toon , a json alternative format. They test LLM accuracy with the format against JSON, (and are generally slightly ahead of JS…
Thanks for sharing this! A question I've grappled with is "how do you make the DOM of a rendered webpage optimal for complex retrieval in both accuracy and tokens?" This could be a really useful transformation to throw in the mix!