Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test-driven development with an LLM for fun and profit

blog.yfzhou.fyi

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Test-driven development with an LLM for fun and profit

#51

Post by czhu12 »

I did something similar for autogenerating RSpec tests in a Rails project.

https://gist.github.com/czhu12/b3fe42454f9fdf626baeaf9c83ab3...

It basically starts from some model or controller, and then parses the Ruby code into an AST, and load all the references, and then parses that code into an AST, up to X number of files, and ships them all off to GPT4-o1 for writing a spec.

I found sometimes, without further prompting, the LLM would write specs that were so heavily mocked that it became almost useless like:

``` mock(add_two_numbers).and_return(3) ... expect(add_two_numbers(1, 2)).to_return(3) ``` (Not that bad, just an illustrating example)

But the tests it generates is quite good overall, and sometimes shockingly good.

Re: Test-driven development with an LLM for fun and profit

#52

Post by sdesol »

energy123 wrote: post #47
sdesol wrote: post #45

Earlier quoted context omitted.

Well given the price $15.00 / 1M input tokens and $60.00 / 1M output* tokens, I would hope so. Given the price, I think it is fair to say it is doing a lot of checks in the background.

It is expensive. But if I'm correct about o1, it means user mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 (or better) models as their daily driver.

> mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1

I think the biggest question is, is o1 scalable. I think o1 does well because it is going back and forth hundreds if not thousands of times. Somebody mentioned in a thread that I was participating in that they let o1 crunch things for 10 minutes. It sounded like it saved them a lot work, so it was well worth it.

Whether or not o1 is practical for the general public is something we will have to wait and see.

Re: Test-driven development with an LLM for fun and profit

#53

Post by energy123 »

sdesol wrote: post #52
energy123 wrote: post #47

Earlier quoted context omitted.

It is expensive. But if I'm correct about o1, it means user mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 (or better) models as their daily driver.

> mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 I think the biggest question is, is o1 scalable. I think o1 does well because it is going back and forth hundreds if not thousands of times. Somebody mentioned in a thread that I was participating in that they let o1 crunch things for 10 minutes. It sounded like it saved them a lot work, so it was well worth it. Whether or…

I'm going to wager "yes" because o3-mini (High) gets equal benchmark scores to o1 despite using 1/3rd as much compute, and because the consistent trend has been towards rapid order-of-magnitude decreases in price for a fixed level of intelligence (trend has many components dovetailing, both hardware and software related). Can't forecast the future, but this would be my bet on a time horizon of < 3 years.

Re: Test-driven development with an LLM for fun and profit

#54

Post by PoppinFreshDo »

steeeeeve wrote: post #24
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

So having clear requirements, a focused purpose for software, and a clear boundary of software responsibility makes for a software development task that can be accomplished? If only people had figured out at some point that the same thing applies when communicating to human software engineers.

If human software engineers refused to work unless those conditions were met, what a wonderful world it would be.

Re: Test-driven development with an LLM for fun and profit

#55

Post by PoppinFreshDo »

Swizec wrote: post #15
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 5. Therefore all new development should be ~~microservices~~ modules written from scratch and interacting via cleanly defined APIs. We figured this out for humans almost 20 years ago. Some really good empirical research. It's the only approach to large scale software development that works. But it requires leadership that gives a shit about the quality of their product and value long-term outcomes over short-term r…

[dead]

Re: Test-driven development with an LLM for fun and profit

#56

Post by vydra »

We implemented something similar for our Java backend project based on my rant here: https://testdriven.com/testdriven-2-0-8354e8ad73d7 Works great! I only look at generated code if it passes the tests. Now, can we use LLMs to generate tests from requirements? Maybe, but tests are mostly declarative and are easier to write than production code most of the time. This approach also allows us to use cheaper models, because the tool will automatically tell the model about compile error and failed tests. Usually, we give it up to five attempts to fix the code.

Re: Test-driven development with an LLM for fun and profit

#57

Post by MarcelOlsz »

whoisnnamdi wrote: post #10
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

This is a helpful breakdown of a trend, thank you Might be a boon for test-driven development. Could turn out that AI coding is the killer app for TDD. I had a similar thought about a year ago but had forgotten, appreciate the reminder

Hey I reached out on twitter to chat :)

Re: Test-driven development with an LLM for fun and profit

#58

Post by ehnto »

sdesol wrote: post #13
smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "Wh…

It's the backwards reasoning that really frustrates me when using LLMs. You ask a question, it says sure do these things, they don't work out and you ask the LLM why not, and it replies yes that thing I told you to do wouldn't work because of these clear reasons.

It would be nice to start at the end of that chain of reasoning instead of the other side.

Another regular example is when it "invents" functions or classes that don't exist, when pressed about them, it will reply of course that won't work, that function doesn't exist.

Okay great, so don't tell me it does with such certainty, is what I would tell a human feeding me imagination as facts all the time. But of course an LLM is not reasoning in the same sense, so this reverse chain of thought is the outcome.

I am finding LLMs far more useful for soft skill topics than engineering type work, simply because of how often it leads me down a path that is eventually a dead end, because of some small detail that was wrong at the very beginning.

Re: Test-driven development with an LLM for fun and profit

#60

Post by willy_k »

ehnto wrote: post #58
sdesol wrote: post #13

Earlier quoted context omitted.

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "Wh…

It's the backwards reasoning that really frustrates me when using LLMs. You ask a question, it says sure do these things, they don't work out and you ask the LLM why not, and it replies yes that thing I told you to do wouldn't work because of these clear reasons. It would be nice to start at the end of that chain of reasoning instead of the other side. Another regular example is when it "invents" functions or classes…

I believe that this is what chain of thought models attempt to address.
Post reply on HN