Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test-driven development with an LLM for fun and profit

blog.yfzhou.fyi

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Test-driven development with an LLM for fun and profit

#21

Post by redcobra762 »

smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

You're going to fall behind eventually, if you continue to treat LLMs with this level of skepticism, as others won't, and the output is accurate enough that it can be useful to improve the efficiency of work in a great many situations.

Rarely are day-to-day written documents (e.g. an email asking for clarification on an issue or to schedule an appointment) of such importance that the occasional error is unforgivable. In situations where a mistake is fatal, yes I would not trust GenAI. But how many of us really work in that kind of a field?

Besides, AI shines when used for creative purposes. Coming up with new ideas or rewording a paragraph for clarity isn't something one does blindly. GenAI is a coworker, not an authority. It'll generate a draft, I may edit that draft or rewrite it significantly, but to preclude it because it could error will eventually slow you down in your field.

Re: Test-driven development with an LLM for fun and profit

#22

Post by iamnotagenius »

smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

Yes, it is good for suumarizing existing text, explaining something or coding; in short any generative/transformative tasks. Not good for information retrieval. Having said that even tiny Qwen 3b/7b coding llms turned out to be very useful in my use experience.

Re: Test-driven development with an LLM for fun and profit

#23

Post by ozten »

agentultra wrote: post #12

This is not a good idea. If you want better tests with more cases exercising your code: write property based tests. Tests form an executable, informal specification of what your software is supposed to do. It should absolutely be written by hand, by a human, for other humans to use and understand. Natural language is not precise enough for even informal specifications of software modules, let alone software systems.…

I got massive productivity gains from having an LLM fill out my test suite.

It is like autocomplete and macros... "Based on these two unit tests, fill out the suite considering b, c, and d. Add any critical corner case tests I have missed or suggest them if they don't fit well."

It is on the human to look at the generated test to ensure a) they are comprehensive and b) useful and c) communicate clearly

Re: Test-driven development with an LLM for fun and profit

#24

Post by steeeeeve »

xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

So having clear requirements, a focused purpose for software, and a clear boundary of software responsibility makes for a software development task that can be accomplished?

If only people had figured out at some point that the same thing applies when communicating to human software engineers.

Re: Test-driven development with an LLM for fun and profit

#25

Post by sitkack »

blopker wrote: post #7

In Rust, there's a controversial practice around putting unit tests in the same file as the actual code. I was put off by it at first, but I'm finding LLM autocomplete is able to be much more effective just being able to see the tests. No clunky loop needed. It's gotten me back into TDD.

If the LLM can't complete a task, you add a test the shows it how to do it. This is multishot incontext learning and programming by example.

As for real TDD, you start with the tests and code until they pass. I haven't used an LLM to do this in Rust yet, but in Python due its dynamic nature, it is much simpler.

You can write the tests, then have the LLM sketch the code out enough so that they pass or at least exist enough to pass a linter. Dev tools are going to feel like magic 18 months from now.

Re: Test-driven development with an LLM for fun and profit

#26

Post by zephraph »

Hey, yeah, this is a fun idea. I built a little toy llm-tdd loop as a Saturday morning side project a little while back: https://github.com/zephraph/llm-tdd.

This doesn't actually work out that well in practice though because the implementations the llm tended to generate were highly specific to pass the tests. There were several times it would cheat and just return hard coded strings that matched the expects of the tests. I'm sure better prompt engineering could help, but it was a fairly funny outcome.

Something I've found more valuable is generating the tests themselves. Obviously you don't wholesale rely on what's generated. Tests can have a certain activation energy just to figure out how to set up correctly (especially if you're in a new project). Having an LLM take a first pass at it and then ensuring it's well structured and testing important codepaths instead of implementation details makes it a lot faster to write tests.

Re: Test-driven development with an LLM for fun and profit

#27

Post by p1necone »

Swizec wrote: post #15
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 5. Therefore all new development should be ~~microservices~~ modules written from scratch and interacting via cleanly defined APIs. We figured this out for humans almost 20 years ago. Some really good empirical research. It's the only approach to large scale software development that works. But it requires leadership that gives a shit about the quality of their product and value long-term outcomes over short-term r…

By large scale do you mean large software or large amounts of developers? Because there's some absolutely massive software in terms of feature set, usefulness and even LoC (not that that is a useful measurement) etc out there made by very small teams.

I'm not sure that you've got the causal relationship the right way around here re: architecture:team size.

Re: Test-driven development with an LLM for fun and profit

#28

Post by lifeisstillgood »

ozten wrote: post #23
agentultra wrote: post #12

This is not a good idea. If you want better tests with more cases exercising your code: write property based tests. Tests form an executable, informal specification of what your software is supposed to do. It should absolutely be written by hand, by a human, for other humans to use and understand. Natural language is not precise enough for even informal specifications of software modules, let alone software systems.…

I got massive productivity gains from having an LLM fill out my test suite. It is like autocomplete and macros... "Based on these two unit tests, fill out the suite considering b, c, and d. Add any critical corner case tests I have missed or suggest them if they don't fit well." It is on the human to look at the generated test to ensure a) they are comprehensive and b) useful and c) communicate clearly

Can you extend that - what was the domain, how did you start? I would like to give this a try but am not quite sure I get it?

Re: Test-driven development with an LLM for fun and profit

#29

Post by emptiestplace »

swatcoder wrote: post #18
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 3. If a problem can be framed in a way that a coding agent can solve... This reminds me of the South Park underwear gnomes. You picked a tool and set an expectation, then just kind of hand wave over the hard part in the middle, as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem. Does it sometimes take 50x effort to understand a problem and the agent well eno…

We've had failed projects since long before LLMs. I think there is a tendency for people to gloss over this (3.) regardless, but working with an LLM it tends to become obvious much more quickly, without investing tens/hundreds of person-hours. I know it's not perfect, but I find a lot of the things people complain about would've been a problem either way - especially when people think they are going to go from 'hello world' to SaaS-billionaire in an hour.

I think mastery of the problem domain is still important, and until we have effectively infinite context windows (that work perfectly), you will need to understand how and when to refactor to maximize quality and relevance of data in context.

Re: Test-driven development with an LLM for fun and profit

#30

Post by __MatrixMan__ »

sdesol wrote: post #14
Arcuru wrote: post #8

Earlier quoted context omitted.

> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs. Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.

Agreed. If the microservice does not provide any value from being isolated, it is just a function call with extra steps.

I think the argument is that the extra value provided is a small enough context window for working with an LLM. Although I'd suggest making it a library if one can manage, that gives you the desired context reduction bounded by interfaces without taking on the complexities of adding an additional microservice.

I imagine throwing a test at an LLM and saying:

> hold the component under test constant (as well as the test itself), and walk the versions of the library until you can tell me where they're compatible and where they break.

If you tried to do that with a git bisect and everything in the same codebase, you'd end up varying all three (test, component, library) which is worse science than holding two constant and varying the third would be.

Post reply on HN