[dead]
Taming LLMs: Using Executable Oracles to Prevent Bad Code
11–20 of 28 posts
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#12"Our goal should be to give an LLM coding agent zero degrees of freedom" Wouldn't that just be called inventing a new language with all the overhead of the languages we already have? Are we getting to the point where getting LLMs to be productive and also write good code is going to require so much overhead and additional procedures and tools that we might as well write the code ourselves. Hmmm...
Actually, no. We always needed good checks - that's why you have techniques like automated canary analysis, extensive testing, checking for coverage - these are forms of "executable oracles". If you wanted to be able to do continuous deployment - you had to be very thorough in your validation. LLMs just take this to the extreme. You can no longer rely on human code reviews (well you can but you give away all the LLM…
If the LLM generates code exactly matching a specification, the specification becomes a conventional programing language. The LLM is just transforming from one language to another.
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#13Earlier quoted context omitted.
Actually, no. We always needed good checks - that's why you have techniques like automated canary analysis, extensive testing, checking for coverage - these are forms of "executable oracles". If you wanted to be able to do continuous deployment - you had to be very thorough in your validation. LLMs just take this to the extreme. You can no longer rely on human code reviews (well you can but you give away all the LLM…
If the LLM generates code exactly matching a specification, the specification becomes a conventional programing language. The LLM is just transforming from one language to another.
Yes, but a programming language with a proverbial sufficiently smart compiler. That is very useful.
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#14Has anyone had success using 2 agents, with one as the creator and one as an adversarial "reviewer"? Is the output usually better or worse?
That works well. Anthropic wrote a writeup on it.
https://www.anthropic.com/engineering/harness-design-long-ru...
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#15Has anyone had success using 2 agents, with one as the creator and one as an adversarial "reviewer"? Is the output usually better or worse?
This is how its meant to be done. Usually with the reviewer being the stronger model.
That said, with both the test driven development this post describes and the reviewer model (its best to do both) you have to provide an escape hatch or out for the model. If you let the model get inescapably stuck with an impossible test or constraints it will just start deleting tests or rewriting the entire codebase in rust or something.
My escape hatch is "expert advice". I let the weak LLM phone a friend when its stuck and ask a smarter LLM for assistance. Its since stopped going crazy and replacing all my tests with gibberish... mostly.
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#16Earlier quoted context omitted.
If the LLM generates code exactly matching a specification, the specification becomes a conventional programing language. The LLM is just transforming from one language to another.
Yes, but a programming language with a proverbial sufficiently smart compiler. That is very useful.
Try writing an exhaustive spec for anything non-trivial and you might see the problem.
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#17[flagged]
Wow, a sudden change in writing style that’s not at all intended to disguise the fact that you’re an llm!
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#18Now is Haskell's time to shine.
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#19"Our goal should be to give an LLM coding agent zero degrees of freedom" Wouldn't that just be called inventing a new language with all the overhead of the languages we already have? Are we getting to the point where getting LLMs to be productive and also write good code is going to require so much overhead and additional procedures and tools that we might as well write the code ourselves. Hmmm...
Zero degrees of freedom is a step too far.
What you want is correctness preserving transformations. Add to this some metrics such as code size, execution speed.
Re: Taming LLMs: Using Executable Oracles to Prevent Bad Code
#20Has anyone had success using 2 agents, with one as the creator and one as an adversarial "reviewer"? Is the output usually better or worse?
[dead]