Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using LLMs to enhance our testing practices

assembled.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Using LLMs to enhance our testing practices

#61

Post by simonw »

mvdtnz wrote: post #49
simonw wrote: post #46

Earlier quoted context omitted.

Which of those did you think were dreck? I think the source code for tools like this one is genuinely good code: https://github.com/simonw/tools/blob/main/extract-urls.html What do you see that's wrong with that?

It's a toy. It doesn't do useful work. The code is fine for the pathetically small sample but that coding style does not scale to real software scales.

The purpose of my "14 things I built in the last week" post was not to demonstrate large software - it was to show how the cost of building small applications has effectively fallen close to zero for me.

I can knock out small but useful applications in genuinely less time than it would take me to Google for an existing solution to the same problem.

You can call them dreck if you like. I call (most of) them useful solutions.

Re: Using LLMs to enhance our testing practices

#62

Post by shadowmanifold »

mastersummoner wrote: post #26

I actually tested Claude Sonnet to see how it would fare at writing a test suite for a background worker. My previous experience was with some version of GPT via Copilot, and it was... not good. I was, however, extremely impressed with Claude this time around. Not only did it do a great job off the bat, but it taught me some techniques and tricks available in the language/framework (Ruby, Rspec) which I wasn't famili…

This latest update to Sonnet is super impressive.

We are really already past the point of being able to discuss these matters though in large groups.

The herd speaks as if all LLMs on all programming languages are basically the same.

It is an absurdity. Talking to the herd is mostly for entertainment at this point. If I actually want to learn something, I will ask Sonnet.

Re: Using LLMs to enhance our testing practices

#63

Post by aoeusnth1 »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

Detecting regressions is the goal. If LLMs can do that for free to cheap, that’s good. It doesn’t have to be complicated.

Re: Using LLMs to enhance our testing practices

#64

Post by GuB-42 »

renegade-otter wrote: post #56
BeetleB wrote: post #36

Earlier quoted context omitted.

Mostly agree. My first thought when I read this post was: Is his goal to test the code, or validate the features? The first problem is he's providing the code, and asking for tests. If his code has a bug, the tests will enshrine those bugs. It's like me writing some code, and then giving it to a junior colleague, not providing any context, and saying "Hey, write some tests for this." This is backwards. I'm not a TDD…

Right! AI is going to help you write passing tests - not BREAK your code, which is the whole point of writing tests.

Tests are not just for breaking your code. Writing passing tests is great for regression testing, which I think is the most important kind of unit testing.

If your goal is to break your code, try fuzzing. For some reason, it seems that the only people who do it are in the field of cybersecurity. Fuzzing can do more than find vulnerabilities.

Re: Using LLMs to enhance our testing practices

#65

Post by idoco »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

Totally agree, especially about the need for well-architected, high-impact tests that go beyond just coverage. At Loadmill, we found out pretty early that building AI to generate tests was just the starting point. The real challenge came with making the system flexible enough to handle complex customer architectures. Think of multiple test environments, unique authentication setups, and dynamic data preparation.

There’s a huge difference between using an LLM to crank out test code and having a product that can actually support complex, evolving setups long-term. A lot of tools work great in demos but don’t hold up for these real-world testing needs.

And yeah, this is even trickier for higher-level tests. Without careful design, it’s way too easy to end up with “dumb” tests that add little real value.

Re: Using LLMs to enhance our testing practices

#66

Post by sdesol »

mvdtnz wrote: post #49
simonw wrote: post #46

Earlier quoted context omitted.

Which of those did you think were dreck? I think the source code for tools like this one is genuinely good code: https://github.com/simonw/tools/blob/main/extract-urls.html What do you see that's wrong with that?

It's a toy. It doesn't do useful work. The code is fine for the pathetically small sample but that coding style does not scale to real software scales.

> style does not scale to real software scales

I think those that dismiss AI completely will fall behind, and those that turn it into a crutch will pay for it in the years to come. I truly believe AI is game changing, as I used it to create standalone functions and get answers that saved me a day or two of research and reading. I've never worked with the cheerio library before but it answered everything I needed to know, among other things. It wasn't perfect though, as it (can't remember the model) wasted some time for me regarding the SQLite library for Node.js.

I think the issue we have right now is we are treating LLM as a final solution (mainly due to investors) instead of thinking of it as a new interface, with quirks that cannot be taken lightly. It's a bit extreme, but I think junior developers should not be allowed to use LLM. LLM is a Power Tool for developers that can easily spot BS and/or have the confidence and knowledge to fix BS that is missed.

Re: Using LLMs to enhance our testing practices

#67

Post by idoco »

dngit wrote: post #34
renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

Great point on focusing on high-impact tests. I agree that LLMs risk giving a false sense of coverage. Maybe a smart strategy is generating boilerplate tests while we focus on custom edge cases.

Absolutely with you on the need for high-impact tests. I find that humans are still way better at coming up with the tests that actually matter, while AI can handle the implementation faster—especially when there’s a human guiding it.

Keeping a human in the loop is essential, in my experience. The AI does the heavy lifting, but we make sure the tests are genuinely useful. That balance helps avoid the trap of churning out “dumb” tests that might look impressive but don’t add real value.

Re: Using LLMs to enhance our testing practices

#68

Post by beepbooptheory »

jeswin wrote: post #44
renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

> Using an autocomplete to "bang it out" seems foolish. Based on my own experience, I find the widespread scepticism on HN about AI-assisted coding misplaced. There will be corner cases, there will be errors, and there will be bugs. There will also be apps for which AI is not helpful at all. But that's fine - nobody is saying otherwise. The question is only about whether it is a _significant_ nett saving on the time…

Ok but looking at those tests for just a second (for createElement), you might want to go through it again, or ask the computer or whatever. For example, edgeCases.test.ts is totally redundant, you are running the same exact tests in children.test.ts.

Edit: such a LLM repo... why did it feel the need to recreate these DOM types? Is your AI just trying to maximize LoC? It just seems like such a pain and potential source of real trouble when these are already available. https://github.com/webjsx/webjsx/blob/main/src%2FjsxTypes.ts

Re: Using LLMs to enhance our testing practices

#69

Post by tsv_ »

Each time a new LLM version comes out, I give it another try at generating tests. However, even with the latest models, tailored GPTs, and well-crafted prompts with code examples, the same issues keep surfacing:

- The models often create several tests within the same equivalence class, which barely expands test coverage

- They either skip parameterization, creating multiple redundant tests, or go overboard with 5+ parameters that make tests hard to read and maintain

- The model seems focused on "writing a test at any cost" often resorting to excessive mocking or monkey-patching without much thought

- The models don’t leverage existing helper functions or classes in the project, requiring me to upload the whole project context each time or customize GPTs for every individual project

Given these limitations, I primarily use LLMs for refactoring tests where IDE isn’t as efficient:

- Extracting repetitive code in tests into helpers or fixtures

- Merging multiple tests into a single parameterized test

- Breaking up overly complex parameterized tests for readability

- Renaming tests to maintain a consistent style across a module, without getting stuck on names

Post reply on HN