Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using LLMs to enhance our testing practices

assembled.com

Post reply on HN
Print view View original

21–30 of 82 posts

Re: Using LLMs to enhance our testing practices

#21

Post by skissane »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

> "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code.

Are there ways we can measure this?

One idea that I’ve had, is collect code coverage separately for each test. If a test isn’t covering any unique code or branches, maybe it is superfluous - although not necessarily, it can make sense to separately test all the boundary conditions of a function, even if doing so doesn’t hit any unique branches.

Maybe prefer a smaller test which covers the same code to a bigger one. However, sometimes if a test is very DRY, it can be more brittle, since it can be non-obvious how to update it to handle a code change. A repetitive test, updating it can be laborious, but at least reasonably obvious how to do so.

Could an LLM evaluate test quality, if you give it a prompt containing some expert advice on good and bad testing practices?

Re: Using LLMs to enhance our testing practices

#22

Post by jazzyjackson »

simonw wrote: post #13
sdesol wrote: post #12

Earlier quoted context omitted.

> 100% agree. We don't expect human developers to be perfect, why should we expect AI assistants. I think the issue is that we are currently being sold that it is. I'm blown away by how useful AI is, and how stupid it can be at the same time. Take a look at the following example: https://app.gitsense.com/?doc=f7419bfb27c896&highlight=&othe... If you click on the sentence, you can see how dumb Sonnet-3.5 and GPT-4 can…

"I think the issue is that we are currently being sold that it is." The message that these systems are flawed appears to be pretty universal to me: ChatGPT footer: "ChatGPT can make mistakes. Check important info." Claude footer: "Claude can make mistakes. Please double-check responses." https://www.meta.ai/ "Messages are generated by AI and may be inaccurate or inappropriate." etc etc etc. I still think the problem…

I appreciate the footer on Kagi Assistant: "Assistant can make mistakes. Think for yourself when using it" - a reminder that theres a tendency to outsource your own train of thought

Re: Using LLMs to enhance our testing practices

#24

Post by taberiand »

apwell23 wrote: post #15

i would love to used to use it change code in ways that compiles and see if test fails. Coverage metric sometimes doesn't really tell you if some piece of code is covered or not.

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea

Re: Using LLMs to enhance our testing practices

#26

Post by mastersummoner »

I actually tested Claude Sonnet to see how it would fare at writing a test suite for a background worker. My previous experience was with some version of GPT via Copilot, and it was... not good.

I was, however, extremely impressed with Claude this time around. Not only did it do a great job off the bat, but it taught me some techniques and tricks available in the language/framework (Ruby, Rspec) which I wasn't familiar with.

I'm certain that it helped having a decent prompt, asking it to consider all the potential user paths and edge cases, and also having a very good understanding of the code myself. Still, this was the first time for me I could honestly say that an LLM actually saved me time as a developer.

Re: Using LLMs to enhance our testing practices

#27

Post by sdesol »

jazzyjackson wrote: post #22
simonw wrote: post #13

Earlier quoted context omitted.

"I think the issue is that we are currently being sold that it is." The message that these systems are flawed appears to be pretty universal to me: ChatGPT footer: "ChatGPT can make mistakes. Check important info." Claude footer: "Claude can make mistakes. Please double-check responses." https://www.meta.ai/ "Messages are generated by AI and may be inaccurate or inappropriate." etc etc etc. I still think the problem…

I appreciate the footer on Kagi Assistant: "Assistant can make mistakes. Think for yourself when using it" - a reminder that theres a tendency to outsource your own train of thought

I would have to imagine 90+ percent of people use LLM and AI to outsource their thought and most will not heed this warning. OpenAI might say "Check important info." but they know most people probably won't do a google search or visit their library to fact check things.

Re: Using LLMs to enhance our testing practices

#28

Post by sdesol »

jayd16 wrote: post #25
nazgul17 wrote: post #23

Should we not, instead, write tests ourselves and have LLMs write the code to make them pass?

Just ask it to do both.

And remember to always challenge the response with both the same and different models. No joke. Just continue the conversation for the example in the blog and ask the LLM "Do you see anything wrong with the code?" and it will spit out "Yes" and explain why.

Re: Using LLMs to enhance our testing practices

#29

Post by nrnrjrjrj »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

There is an art to writing tests especially getting absraction levels right. For example do you integration test hitting the password field with 1000 cases or do that as a unit test, and does doing it as a unit test sufficiently cover this.

AI could do all this thinking in the future but not yet I believe!

Let alone the codebase is likely a mess of bad practice already (never seen one that isn't! That is life) so often part of the job is leaving the campground a bit better than how you found it.

LLMs can help now on last mile stuff. Fill in this one test. Generate data for 100 test cases. Etc.

Re: Using LLMs to enhance our testing practices

#30

Post by fijiaarone »

skissane wrote: post #21
renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

> "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Are there ways we can measure this? One idea that I’ve had, is collect code coverage separately for each test. If a test isn’t covering any unique code or branches, maybe it is superfluous - although not necessarily, it can make sense to separately test…

Sometimes you actually have to think, or hire someone who can. Go join the comments section on the Goodharts Law post to go on about measuring magical metrics.
Post reply on HN