Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using LLMs to enhance our testing practices

assembled.com

Post reply on HN
Print view View original

31–40 of 82 posts

Re: Using LLMs to enhance our testing practices

#31

Post by skissane »

fijiaarone wrote: post #30
skissane wrote: post #21

Earlier quoted context omitted.

> "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Are there ways we can measure this? One idea that I’ve had, is collect code coverage separately for each test. If a test isn’t covering any unique code or branches, maybe it is superfluous - although not necessarily, it can make sense to separately test…

Sometimes you actually have to think, or hire someone who can. Go join the comments section on the Goodharts Law post to go on about measuring magical metrics.

> Sometimes you actually have to think, or hire someone who can.

I'm perfectly capable of thinking. Thinking about "how can I create a system which reduces some of my cognitive load on testing so I can spend more of my cognitive resources on other things" is a particularly valuable form of thinking.

> Go join the comments section on the Goodharts Law post to go on about measuring magical metrics.

That problem is when managers take a metric and turn it into a KPI. That doesn't happen to all metrics. I can think of many metrics I've personally collected that no manager ever once gazed upon.

The real measure of a metric's value, is how meaningful a domain expert finds it to be. And if the answer to that is "not very" – is that an inherent property of metrics, or a sign that the metric needs to be refined?

Re: Using LLMs to enhance our testing practices

#32

Post by mvdtnz »

hitradostava wrote: post #9
simonw wrote: post #7

Earlier quoted context omitted.

The answer to this is code review. If an LLM writes code for you - be it implementation or tests - you review it before you land it. If you don't understand how the code works, don't approve it. Sure, complacent developers will get burned. They'll find plenty of other non-AI ways to burn themselves too.

100% agree. We don't expect human developers to be perfect, why should we expect AI assistants. Code going to production should go through review. I do think that LLMs will increase the volume of bad code though. I use Cursor a lot, and occasionally it will produce perfect code, but often I need to direct and refine, and sometimes throw away. But I'm sure many devs will get lazy and just push once they've got the thi…

> We don't expect human developers to be perfect, why should we expect AI assistants.

What absolute nonsense. What an absurd false equivalence. It's not that we expect perfection or even human level performance from "AI". It's that the crap that comes out of LLMs is not even at the level of a first year student. I've never in my entire life reviewed the code of a junior engineer and seen them invent third party APIs from whole cloth. I've never had a junior send me code that generates a payload that doesn't validate at the first layer of the operation with zero manual testing to check it. No junior has ever asked me to review a pull request containing references to an open source framework that doesn't exist anywhere in my application. Yet these scenarios are commonplace in "AI" generated code.

Re: Using LLMs to enhance our testing practices

#33

Post by jaredsohn »

skissane wrote: post #31
fijiaarone wrote: post #30

Earlier quoted context omitted.

Sometimes you actually have to think, or hire someone who can. Go join the comments section on the Goodharts Law post to go on about measuring magical metrics.

> Sometimes you actually have to think, or hire someone who can. I'm perfectly capable of thinking. Thinking about "how can I create a system which reduces some of my cognitive load on testing so I can spend more of my cognitive resources on other things" is a particularly valuable form of thinking. > Go join the comments section on the Goodharts Law post to go on about measuring magical metrics. That problem is when…

Good tests reduce your cognitive load; you can have more confidence that code will work and spend less time worrying that someone will break it.

BTW, I think above are the best metrics to use for tests. Actually measuring it can be hard, but I think keeping track of when functionality doesn't work and people break your code is a good start.

And I think all of this should be measured in terms of doing the right thing business logic-wise and weighing importance of what needs testing based on the business value of when things don't work.

Re: Using LLMs to enhance our testing practices

#34

Post by dngit »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

Great point on focusing on high-impact tests. I agree that LLMs risk giving a false sense of coverage. Maybe a smart strategy is generating boilerplate tests while we focus on custom edge cases.

Re: Using LLMs to enhance our testing practices

#35

Post by mkleczek »

I am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself.

In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code.

IOW: how should we treat generated code?

If we treat it in the same way as code produced by humans (ie. we maintain it) then the maintenance cost grows (super-linearly) with the amount of code we generate. To make matters worse for LLM: since the code it generates is buggy it means we have more buggy code to maintain. Code review is not the answer because code review power in finding bugs is very weak.

This is unlike compilers (that also generate code) because we don't maintain code generated by compilers - we regenerate it anytime we need.

The fundamental issue is: for a given set of requirements the goal is to produce less code, not more. _Any_ code generation (however smart it might be) goes against this goal.

EDIT: typos

Re: Using LLMs to enhance our testing practices

#36

Post by BeetleB »

renegade-otter wrote: post #14

In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…

Mostly agree.

My first thought when I read this post was: Is his goal to test the code, or validate the features?

The first problem is he's providing the code, and asking for tests. If his code has a bug, the tests will enshrine those bugs. It's like me writing some code, and then giving it to a junior colleague, not providing any context, and saying "Hey, write some tests for this."

This is backwards. I'm not a TDD guy, but you should think of your test cases independent of your code.

Re: Using LLMs to enhance our testing practices

#37

Post by mvdtnz »

mkleczek wrote: post #35

I am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself. In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code. IOW: how should we treat generated code? If we…

You should NEVER modify generated code. All of our generated code is pretended with a big comment that says "GENERATED CODE DO NOT MODIFY. This code could be regenerated at any time and any changes will be lost."

If you need to change behaviour of generated code you need to change your generator to provide the right hooks.

Obviously none of this applies to "AI" generated code because the "AI" generator is not deterministic and will hallucinate different bugs from run to run. You must treat "AI" generated code as if it was written by the dumbest person you've ever worked with.

Re: Using LLMs to enhance our testing practices

#38

Post by mkleczek »

mvdtnz wrote: post #37
mkleczek wrote: post #35

I am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself. In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code. IOW: how should we treat generated code? If we…

You should NEVER modify generated code. All of our generated code is pretended with a big comment that says "GENERATED CODE DO NOT MODIFY. This code could be regenerated at any time and any changes will be lost." If you need to change behaviour of generated code you need to change your generator to provide the right hooks. Obviously none of this applies to "AI" generated code because the "AI" generator is not determi…

That's exactly my point :)

Re: Using LLMs to enhance our testing practices

#39

Post by simonw »

mvdtnz wrote: post #32
hitradostava wrote: post #9

Earlier quoted context omitted.

100% agree. We don't expect human developers to be perfect, why should we expect AI assistants. Code going to production should go through review. I do think that LLMs will increase the volume of bad code though. I use Cursor a lot, and occasionally it will produce perfect code, but often I need to direct and refine, and sometimes throw away. But I'm sure many devs will get lazy and just push once they've got the thi…

> We don't expect human developers to be perfect, why should we expect AI assistants. What absolute nonsense. What an absurd false equivalence. It's not that we expect perfection or even human level performance from "AI". It's that the crap that comes out of LLMs is not even at the level of a first year student. I've never in my entire life reviewed the code of a junior engineer and seen them invent third party APIs…

That problem genuinely doesn't matter to me at all.

If an LLM hallucinates a method that doesn't exist I find out the moment I try and run the code.

If I'm using ChatGPT Code Interpreter (for Python) or Claude analysis mode (for JavaScript) I don't even have to intervene: the LLM can run in a loop, generating code, testing that it executes without errors and correcting any mistakes it makes.

I still need to carefully review the code, but the mistakes which cause it not to run at all are by far the least amount of work to identify.

Re: Using LLMs to enhance our testing practices

#40

Post by DanHulton »

simonw wrote: post #13
sdesol wrote: post #12

Earlier quoted context omitted.

> 100% agree. We don't expect human developers to be perfect, why should we expect AI assistants. I think the issue is that we are currently being sold that it is. I'm blown away by how useful AI is, and how stupid it can be at the same time. Take a look at the following example: https://app.gitsense.com/?doc=f7419bfb27c896&highlight=&othe... If you click on the sentence, you can see how dumb Sonnet-3.5 and GPT-4 can…

"I think the issue is that we are currently being sold that it is." The message that these systems are flawed appears to be pretty universal to me: ChatGPT footer: "ChatGPT can make mistakes. Check important info." Claude footer: "Claude can make mistakes. Please double-check responses." https://www.meta.ai/ "Messages are generated by AI and may be inaccurate or inappropriate." etc etc etc. I still think the problem…

But on the other other hand, there's the commercials generated to sell new models or new model features, that FREQUENTLY lie about actual capabilities and fake demos and don't actually end with an equivalent amount of time going over how actual usage may be shit and completely unlike the advertisement.

I'd say parent is absolutely correct - we ARE being sold (quite literally, through promotional material, i.e. ads) that these models are way more capable than they actually are.

Post reply on HN