Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using LLMs to enhance our testing practices

assembled.com

Post reply on HN
Print view View original

51–60 of 82 posts

Re: Using LLMs to enhance our testing practices

#51

Post by DeathArrow »

It's hard to generate tests for typical C# code. Or for any context where you have external dependencies.

If you have injected services in your current service, the LLM doesn't know anything about those so it makes poor guesses. You have to bring those in context, so they can be mocked properly.

You end up spending a lot of time guiding the LLM, so it's not measurably faster than writing test by hand.

I want my prompt to be: "write unit tests for XYZ method" without having to accurately describe it the prompt what the method does, how it does it and why it does it. Writing too many details in the prompt takes the same time as writing the code myself.

Github Copilot should be better since it's supposed to have access to you entire code base. But somehow it doesn't look at dependencies and it just uses the knowledge of the codebase for stylistic purposes.

It's probably my fault, there are for sure better ways to use LLMs for code, but I am probably not the only one who struggles.

Re: Using LLMs to enhance our testing practices

#53

Post by danmaz74 »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

I subscribe to the concept of the "pyramid of tests" - lots of simpler unit tests, fewer integration tests, and very few end-to-end tests. I find that using LLMs to write unit tests is very useful. If I just wrote code which has good naming both for the classes, methods and variables, useful comments where necessary and if I already have other tests which the LLMs can use as examples for how I test things, I usually just need to read the created tests and sometimes add some test cases, just writing the "it should 'this and that'" part for cases which weren't covered.

An added bonus is that if the tests aren't what you expect, often it helps you understand that the code isn't as clear as it should be.

Re: Using LLMs to enhance our testing practices

#54

Post by smokel »

mkleczek wrote: post #35

I am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself. In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code. IOW: how should we treat generated code? If we…

I agree. Adding unit tests without a good reason comes at a cost.

Refactoring is harder, especially if it's not clear why a test is in place. I've seen many developers disable tests simply because they could not understand how, or why, to fix them.

I'm hopeful that LLMs can provide guidance in removing useless tests or simplifying things. In an ideal future they may even help in formulating requirements or design documentation.

Re: Using LLMs to enhance our testing practices

#55

Post by NitpickLawyer »

mvdtnz wrote: post #49
simonw wrote: post #46

Earlier quoted context omitted.

Which of those did you think were dreck? I think the source code for tools like this one is genuinely good code: https://github.com/simonw/tools/blob/main/extract-urls.html What do you see that's wrong with that?

It's a toy. It doesn't do useful work. The code is fine for the pathetically small sample but that coding style does not scale to real software scales.

|_____| ... ... ... |_____| I guess we all see where the goalposts will be tomorrow. Good code, good problem, I don't like the language. Or something :)

Re: Using LLMs to enhance our testing practices

#56

Post by renegade-otter »

BeetleB wrote: post #36
renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

Mostly agree. My first thought when I read this post was: Is his goal to test the code, or validate the features? The first problem is he's providing the code, and asking for tests. If his code has a bug, the tests will enshrine those bugs. It's like me writing some code, and then giving it to a junior colleague, not providing any context, and saying "Hey, write some tests for this." This is backwards. I'm not a TDD…

Right! AI is going to help you write passing tests - not BREAK your code, which is the whole point of writing tests.

Re: Using LLMs to enhance our testing practices

#57

Post by mkleczek »

smokel wrote: post #54
mkleczek wrote: post #35

I am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself. In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code. IOW: how should we treat generated code? If we…

I agree. Adding unit tests without a good reason comes at a cost. Refactoring is harder, especially if it's not clear why a test is in place. I've seen many developers disable tests simply because they could not understand how, or why, to fix them. I'm hopeful that LLMs can provide guidance in removing useless tests or simplifying things. In an ideal future they may even help in formulating requirements or design doc…

> I'm hopeful that LLMs can provide guidance in removing useless tests or simplifying things. In an ideal future they may even help in formulating requirements or design documentation

I am very sceptical here as well. The biggest problem with formulating requirements or design documentation is translation from informal to formal language. In other words... writing programs.

LLMs are good at generating content that doesn't provide useful information (ie. have low information content). Their usefulness right now is caused by the fact that people are used to reading lot of text and distill information from it (ie. all the useless e-mails formulated in corporate language, all multi-page requirement documents formulated in human readable form). The job of a software engineer is to extract information from low information content text and write it down in a formal language.

In this context:

What I expect in the long run is that people will start to value high information content and concise text. And obviously - it cannot be generated by any LLM, because LLM cannot provide any information by itself. There is really no point in: provide short high information content text (ie. prompt) to LLM -> receive long low information content text from LLM -> extract information from long text.

Re: Using LLMs to enhance our testing practices

#58

Post by rgmerk »

taberiand wrote: post #24
apwell23 wrote: post #15

i would love to used to use it change code in ways that compiles and see if test fails. Coverage metric sometimes doesn't really tell you if some piece of code is covered or not.

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea

LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.

Re: Using LLMs to enhance our testing practices

#59

Post by thanksgiving »

jeswin wrote: post #44
renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

> Using an autocomplete to "bang it out" seems foolish. Based on my own experience, I find the widespread scepticism on HN about AI-assisted coding misplaced. There will be corner cases, there will be errors, and there will be bugs. There will also be apps for which AI is not helpful at all. But that's fine - nobody is saying otherwise. The question is only about whether it is a _significant_ nett saving on the time…

I want to bring my own experience from a code base I briefly worked on, I worked on a module of code where basically all the unit tests assertions were commented out. This was about ten years ago. The meta is there should still be someone responsible for the code an LLM generated and there should still be at least one more person who does a decent code review at some point. Otherwise, the unit tests being there is useless just like the example I gave on top with the assertions removed.

Re: Using LLMs to enhance our testing practices

#60

Post by holbrad »

danmaz74 wrote: post #53
renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

I subscribe to the concept of the "pyramid of tests" - lots of simpler unit tests, fewer integration tests, and very few end-to-end tests. I find that using LLMs to write unit tests is very useful. If I just wrote code which has good naming both for the classes, methods and variables, useful comments where necessary and if I already have other tests which the LLMs can use as examples for how I test things, I usually…

I also subscribe to a testing pyramid but I think it's commonly upside down IMO.

You should have a few very granular unit tests for where they make the most sense (Known dangerous areas, or where they are very easy to write eg. analysis)

More library/service tests. I read in an old config file and it has the values I expect.

Integration/system tests should be the most common, I spin up the entire app in a container and use the public API to test the application as a whole.

Then most importantly automated UI tests, I do the standard normal customer workflows and either it works or it doesn't.

The nice thing is that when you strongly rely on UI and public API tests you can have very strong confidence that your core features actually work. And when there are bugs they are far more niche. And this doesn't require many tests at all.

(We've all been in the situation where the 50,000 unit tests pass and the application is critically broken)

Post reply on HN