Super interesting approach! We've been working on the opposite - always getting your Unit tests written with every PR. The idea is that you don't have to bother running or writing them, you just get them delivered in your Github repo. You can check it out here https://www.codebeaver.ai
Test-driven development with an LLM for fun and profit
71–80 of 91 posts
Re: Test-driven development with an LLM for fun and profit
#72Super interesting approach! We've been working on the opposite - always getting your Unit tests written with every PR. The idea is that you don't have to bother running or writing them, you just get them delivered in your Github repo. You can check it out here https://www.codebeaver.ai
First, I'm a fan of LLMs reducing friction in tests, but I would be concerned with the false sense of confidence here. The demo gif shows "hey I wrote your tests and they pass, go ahead and merge them"
OP makes a valid point
> Now we contend with the “who guards the guard” problem. Because LLMs are unreliable agents, it might so happen that Claude just scammed us by spitting out useless (or otherwise low-effort) test cases. [...] So it’s time to introduce some human input in the form of additional test cases, which is made extra convenient since the model already provided the overall structure of our test. If those cases pass, we can be reasonably confident in integrating this function into our codebase.
In our repos, I would love to have an LLM tool/product that helps out with test writing, but the workflow certainly needs to have some human in the loop for the time being. More like "Here I got you started with test coverage, add a few more of your own" or "Give me a few bullet points of cases that should pass or fail" and review the test code, not "go ahead and merge these tests I wrote for you"
Re: Test-driven development with an LLM for fun and profit
#73One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…
> Coding assistants based on o1 and Sonnet are pretty great at coding with I had a very similar impression (wrote more in https://hua.substack.com/p/are-longer-context-windows-all-yo...).
One framing is that effective context window (i.e. the length that the model is able to effectively reason over) determines how useful the model is. A human new grad programmer might effectively reason over 100s or 1000s of tokens but not millions - which is why we carefully scope the work and explain where to look for relevant context only. But a principal engineer might reason over many many millions of context - code yes, but also organizational and business context.
Trying to carefully select those 50k tokens is extremely difficult for LLMs/RAG today. I expect models to get much longer effective context windows but there are hardware / cost constraints which make this more difficult.
Re: Test-driven development with an LLM for fun and profit
#74One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…
> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs. Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.
I wonder if we'll see a return of the kind of interface file present in C++, Ocaml, and Ada. These files, well commented, are naturally the context window to use for reference for a module.
Even if languages don't grow them back as a first class feature, some format that is auto generated from the code and doesn't include the function bodies is really what is needed here.
Re: Test-driven development with an LLM for fun and profit
#75Super interesting approach! We've been working on the opposite - always getting your Unit tests written with every PR. The idea is that you don't have to bother running or writing them, you just get them delivered in your Github repo. You can check it out here https://www.codebeaver.ai
Test driven development is sequenced the way it is for a reason. Getting a failing test first builds confidence that the test is, you know, actually testing something. And the process of writing the tests is often where the largest amount of reasoning about design choices takes place.
Having an LLM generate the tests after you've already written the code for them is super counterproductive. Who knows whether those tests actually test anything?
I know this gets into "I wanted AI to do my laundry, not my art" territory, but a far more rational division of labor is for the humans to write the tests (maybe with the assistance of an autocomplete model) and give those as context for the AI. Humans are way better at thinking of edge cases and design constraints than the models are at this point in the game.
Re: Test-driven development with an LLM for fun and profit
#76Earlier quoted context omitted.
> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs. Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.
I wonder if we'll see a return of the kind of interface file present in C++, Ocaml, and Ada. These files, well commented, are naturally the context window to use for reference for a module. Even if languages don't grow them back as a first class feature, some format that is auto generated from the code and doesn't include the function bodies is really what is needed here.
Python (which I mention because it is the preferred language of LLM output) has grown stub files that would work for this:
https://peps.python.org/pep-0484/#stub-files
I guess that this usecase would be an argument to include docstrings in your Python stub files, which I hadn’t considered before.
Re: Test-driven development with an LLM for fun and profit
#77Earlier quoted context omitted.
I think the argument is that the extra value provided is a small enough context window for working with an LLM. Although I'd suggest making it a library if one can manage, that gives you the desired context reduction bounded by interfaces without taking on the complexities of adding an additional microservice. I imagine throwing a test at an LLM and saying: > hold the component under test constant (as well as the tes…
> I think the argument is that the extra value provided is a small enough context window for working with an LLM. I'm not sure moving something that could work as function to a microservice would save much context. If anything, I think you are adding more context, since you would need to talk about the endpoint and having it route to the function that does what you need. When it is all over, you need to describe what…
Oh certainly. I was arguing that if you need more isolation than a function gives you, don't jump to the conclusion that you need a service. Consider a library as a middle ground.
Re: Test-driven development with an LLM for fun and profit
#78One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…
> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs. Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.
Yeah, I think monorepos will be better for LLMs. Easier to refactor module boundaries as context grows or requirements change.
But practices like stronger module boundaries, module docs, acceptance tests on internal dev-facing module APIs, etc are all things that will be much more valuable for LLM consumption. (And might make things more pleasant for humans too!)
Re: Test-driven development with an LLM for fun and profit
#79In Rust, there's a controversial practice around putting unit tests in the same file as the actual code. I was put off by it at first, but I'm finding LLM autocomplete is able to be much more effective just being able to see the tests. No clunky loop needed. It's gotten me back into TDD.
The benefit of this approach is that you can directly test any function in the same scope without altering its visibility: it implicitly encourages you to test all functions (and design functions in a way they can be tested, as you are writing tests as you write code), not just those part of the public api contract.
Plus you can update tests, code, and comments in one go, with visibility into them at all times.
Re: Test-driven development with an LLM for fun and profit
#80I’m not going to claim I’ve solved this and figured out “the way” to use LLMs for tests, but I’ve found that copy-and-pasting code + tests and then providing a short essay about my own reasoning of edge cases followed with something along the lines of
“your job is to find out what edge cases my reasoning isn’t accounting for, cases that would expose latent properties of the implementation not exposed via its contract, cases tested for by other similar code, domain exceptions I’m not accounting for, cases that test unexplored code paths, cases that align exactly with chunking boundaries or that break chunking assumptions, or any other edge cases I’m neglecting to mention that would be useful both to catch mistakes in the current code and to handle foreseeable mistakes that could arise from refactoring in the future. Try to understand how the existing test cases are defined to catch possibly problematic inputs and extend accordingly. Take into account both the api contract and the underlying implementation and approach this matter from an adversarial perspective where the goal of the tests is to challenge the author’s assumptions and break their code” has been useful.