Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test-driven development with an LLM for fun and profit

blog.yfzhou.fyi

Post reply on HN
Print view View original

31–40 of 91 posts

Re: Test-driven development with an LLM for fun and profit

#31

Post by ozten »

lifeisstillgood wrote: post #28
ozten wrote: post #23

Earlier quoted context omitted.

I got massive productivity gains from having an LLM fill out my test suite. It is like autocomplete and macros... "Based on these two unit tests, fill out the suite considering b, c, and d. Add any critical corner case tests I have missed or suggest them if they don't fit well." It is on the human to look at the generated test to ensure a) they are comprehensive and b) useful and c) communicate clearly

Can you extend that - what was the domain, how did you start? I would like to give this a try but am not quite sure I get it?

Backend coding for web services.

In the past I would hand write 8 or 9 unit tests. Now I write the first one or two and then brain dump anything else into the LLM prompt. It then outputs mine plus 6 or more.

I delete any that seem low value or ridiculous or have a follow up prompt to ask for refinements. Then just copy/pasta back into the codebase out of the chat.

Re: Test-driven development with an LLM for fun and profit

#32

Post by dingnuts »

emptiestplace wrote: post #29
swatcoder wrote: post #18

Earlier quoted context omitted.

> 3. If a problem can be framed in a way that a coding agent can solve... This reminds me of the South Park underwear gnomes. You picked a tool and set an expectation, then just kind of hand wave over the hard part in the middle, as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem. Does it sometimes take 50x effort to understand a problem and the agent well eno…

We've had failed projects since long before LLMs. I think there is a tendency for people to gloss over this (3.) regardless, but working with an LLM it tends to become obvious much more quickly, without investing tens/hundreds of person-hours. I know it's not perfect, but I find a lot of the things people complain about would've been a problem either way - especially when people think they are going to go from 'hello…

well according to xianshou's profile they work in finance so it makes sense to me that they would gloss over the hard part of programming when describing how AI is going to improve it

Re: Test-driven development with an LLM for fun and profit

#33

Post by phaedrus »

xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

50K context is an interesting number because I think there's a lot to explore with software within an order of magnitude that size. With apologies to Richard Feynman, I call it, "There's plenty of room in the middle." My idea there is the rapid expansion of computing power during the reign of Moore's law left the design space of "medium sized" programs under-explored. These would be programs in the range of 100's of kilobytes to low megabytes.

Re: Test-driven development with an LLM for fun and profit

#34

Post by horsawlarway »

sdesol wrote: post #13
smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "Wh…

Isn't this essentially making the point of the post above you?

For comparison - if I just do a web search for "Did homer simpson go to mars" I get immediately linked to the wikipedia page for that exact episode (https://en.wikipedia.org/wiki/The_Marge-ian_Chronicles), and the plot summary is less to read than your LLM output - It clearly summarizes that Marge & Lisa (note - NOT homer) almost went to mars, but did not go. Further - the summary correctly includes the outro which does show Marge and Lisa on mars in the year 2051.

Basically - for factual content, the LLM output was a garbage game of telephone.

Re: Test-driven development with an LLM for fun and profit

#35

Post by energy123 »

smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

We need a hallucination benchmark.

My experience is, o1 is very good at avoiding hallucinations and I trust it more, but o1-mini and 4o are awful.

Re: Test-driven development with an LLM for fun and profit

#37

Post by Swizec »

p1necone wrote: post #27
Swizec wrote: post #15

Earlier quoted context omitted.

> 5. Therefore all new development should be ~~microservices~~ modules written from scratch and interacting via cleanly defined APIs. We figured this out for humans almost 20 years ago. Some really good empirical research. It's the only approach to large scale software development that works. But it requires leadership that gives a shit about the quality of their product and value long-term outcomes over short-term r…

By large scale do you mean large software or large amounts of developers? Because there's some absolutely massive software in terms of feature set, usefulness and even LoC (not that that is a useful measurement) etc out there made by very small teams. I'm not sure that you've got the causal relationship the right way around here re: architecture:team size.

What does team size have to do with this? Small teams can (and should) absolutely build modularized software ...

You simply cannot build a [working/maintainable] large piece of software if everything is connected to everything and any one change may cause issues in conceptually unrelated pieces of code. As soon as your codebase is bigger than what you can fully memorize, you need modules, separation of concerns, etc.

Re: Test-driven development with an LLM for fun and profit

#38

Post by ziddoap »

dingnuts wrote: post #32
emptiestplace wrote: post #29

Earlier quoted context omitted.

We've had failed projects since long before LLMs. I think there is a tendency for people to gloss over this (3.) regardless, but working with an LLM it tends to become obvious much more quickly, without investing tens/hundreds of person-hours. I know it's not perfect, but I find a lot of the things people complain about would've been a problem either way - especially when people think they are going to go from 'hello…

well according to xianshou's profile they work in finance so it makes sense to me that they would gloss over the hard part of programming when describing how AI is going to improve it

Working in one domain does not preclude knowledge of others. I work in cybersec but spent my first working decade in construction estimation for institutional builds. I can talk confidently about firewalls or the hospital you want to build.

No need to make assumptions based on a one-line hacker news profile.

Re: Test-driven development with an LLM for fun and profit

#39

Post by hitchstory »

swatcoder wrote: post #18
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 3. If a problem can be framed in a way that a coding agent can solve... This reminds me of the South Park underwear gnomes. You picked a tool and set an expectation, then just kind of hand wave over the hard part in the middle, as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem. Does it sometimes take 50x effort to understand a problem and the agent well eno…

>as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem.

It's not, but if you can A) make it cheap to try out different types of framings - not all of them have to work and B) automate everything else then the labor intensity of programming decreases drastically.

>At commercial quality, is it an accessible skill for inexperienced people

I'd expect the opposite, it would be an extremely inaccessible skill requiring high skill and high pay. But, if 2 people can deliver as much as 15 people at a higher quality and they're paid triple, it's still way cheaper overall.

I would still expect somebody following this development pattern to routinely discover a problem the LLM can't deal with and have to dive under the hood to fix it - digging down below multiple levels of abstraction. This would be Hard with a capital H.

Re: Test-driven development with an LLM for fun and profit

#40

Post by feznyng »

zephraph wrote: post #26

Hey, yeah, this is a fun idea. I built a little toy llm-tdd loop as a Saturday morning side project a little while back: https://github.com/zephraph/llm-tdd . This doesn't actually work out that well in practice though because the implementations the llm tended to generate were highly specific to pass the tests. There were several times it would cheat and just return hard coded strings that matched the expects of the…

Did you show the test cases to it? Maybe blinding it would solve the tailoring problem.
Post reply on HN