Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test-driven development with an LLM for fun and profit

blog.yfzhou.fyi

Post reply on HN
Print view View original

11–20 of 91 posts

Re: Test-driven development with an LLM for fun and profit

#11

Post by eesmith »

> recognize and parse any and all ipv4 and ipv6 addresses and CIDRs contained within it (these may be surrounded by random words or symbols like commas), then return them as a list'

Did I miss the generated code and test cases? I would like to see how complete it was.

For example, for IPv4 does it only handle quad-dotted IP addresses, or does it also handle decimal and hex formats?

For that matter, should it handle those, and if so, where there clarification of what exactly 'all ipv4 ... addresses' means?

I can think of a lot of tricky cases (like 1.2.3.4.5 and 3::2::1 as invalid cases, or http://[2001:db8:4006:812::200e] to test for "symbols like commas"), and would like to see if the result handles them.

Re: Test-driven development with an LLM for fun and profit

#12

Post by agentultra »

This is not a good idea.

If you want better tests with more cases exercising your code: write property based tests.

Tests form an executable, informal specification of what your software is supposed to do. It should absolutely be written by hand, by a human, for other humans to use and understand. Natural language is not precise enough for even informal specifications of software modules, let alone software systems.

If using LLM's to help you write the code is your jam, I can't stop you, but at least write the tests. They're more important.

As an aside, I understand how this antipathy towards TDD develops. People write unit tests, after writing the implementation, because they see it as boilerplate code that mirrors what the code they're testing already does. They're missing the point of what makes a good test useful and sufficient. I would not expect generating more tests of this nature is going to improve software much.

Edit added some wording for clarity

Re: Test-driven development with an LLM for fun and profit

#13

Post by sdesol »

smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output?

I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong.

This isn't foolproof though, but it does provide some level of confidence in the answer.

Here is a quick example in which I analyze results from multiple LLMs that answered, "When did Homer Simpson go to Mars?"

https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abf...

If you look at the yes and no table, all except GPT-4o and GPT-4o mini said no. After asking GPT-4o who was correct, it provided "evidence" on an episode so I asked for more information on that episode. Based on what it said, it looks like the mission to Mars was a hoax and when I challenged GPT-4o on this, it agreed and said Homer never went to Mars, like others have said.

I then asked Sonnet 3.5 about the episode and it said GPT-4o misinterpreted the plot.

https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abf...

At this point, I am confident (but not 100% sure) Homer never went to Mars and if I really needed to know, I'll need to search the web.

Re: Test-driven development with an LLM for fun and profit

#14

Post by sdesol »

Arcuru wrote: post #8
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs. Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.

Agreed. If the microservice does not provide any value from being isolated, it is just a function call with extra steps.

Re: Test-driven development with an LLM for fun and profit

#15

Post by Swizec »

xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 5. Therefore all new development should be ~~microservices~~ modules written from scratch and interacting via cleanly defined APIs.

We figured this out for humans almost 20 years ago. Some really good empirical research. It's the only approach to large scale software development that works.

But it requires leadership that gives a shit about the quality of their product and value long-term outcomes over short-term rewards.

Re: Test-driven development with an LLM for fun and profit

#16

Post by sdesol »

xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> you can speed up development by 10x.

If you know what you are doing, then yes. If you are a domain expert and can articulate your thoughts clearly in a prompt, you will most likely see a boost—perhaps two to three times—but ten times is unlikely. And if you don't fully understand the problem, you may experience a negative effect.

Re: Test-driven development with an LLM for fun and profit

#17

Post by Marceltan »

smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

Agree. My biggest pain point with LLM code review tools is that they sometimes add 40 comments for a PR changing 100 lines of code. Gets noisy and hard to decipher what really matters.

Along the lines of verifiability, my take is that running a comprehensive suite of tests in CI/CD is going to be table stakes soon given that LLMs are only going to be contributing more and more code.

Re: Test-driven development with an LLM for fun and profit

#18

Post by swatcoder »

xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 3. If a problem can be framed in a way that a coding agent can solve...

This reminds me of the South Park underwear gnomes. You picked a tool and set an expectation, then just kind of hand wave over the hard part in the middle, as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem.

Does it sometimes take 50x effort to understand a problem and the agent well enough to get that done? Are there classes of problems where it can't be done? Are either of those concerns something you can recognize before they impact you? At commercial quality, is it an accessible skill for inexperienced people or do you need a mastery of coding, the problem domain, or the coding agent to be able to rely on it? Can teams recruit people who can reliable achieve any of this? How expensive is that talent? etc

Re: Test-driven development with an LLM for fun and profit

#19

Post by throwup238 »

sdesol wrote: post #16
xianshou wrote: post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> you can speed up development by 10x. If you know what you are doing, then yes. If you are a domain expert and can articulate your thoughts clearly in a prompt, you will most likely see a boost—perhaps two to three times—but ten times is unlikely. And if you don't fully understand the problem, you may experience a negative effect.

I think it also depends on how much yak-shaving is involved in the domain, regardless of expertise. Whether that’s something simple like remembering the right bash incantation or something more complex like learning enough Terraform and providers to be able to spin up cloud infrastructure.

Some projects just have a lot of stuff to do around the edges and LLMs excel at that.

Re: Test-driven development with an LLM for fun and profit

#20

Post by manmal »

sdesol wrote: post #13
smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "Wh…

Relatedly, asking LLMs what happens in a TV episode, or a series in general, I usually get very low quality and mostly flat out wrong answers. That baffles me, as I thought there are multiple well structured synopses for any TV series in the training data.
Post reply on HN