Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Semantic unit testing: test code without executing it

alexmolas.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Semantic unit testing: test code without executing it

#31

Post by IshKebab »

RainyDayTmrw wrote: post #15

I'm skeptical. Most of us maintaining medium sized codebases or larger are constantly fighting nondeterminism in the form of flaky tests. I can't imagine choosing a design that starts with nondeterminism baked in. And if you're really dead-set on paying nondeterminism to get more coverage, property-based testing has existed for a long time and has a comparatively solid track record.

I agree. I want this as a code review tool to check if people forgot to update comments - "it looks like this now adds instead of multiplies, but the comment says otherwise; did you forget to update it?".

Seems of dubious value as unit tests. LLMs don't seem to be quite smart enough for that in my experience, unless your bugs are really as trivial as adding instead of multiplying, in which case god help you.

Re: Semantic unit testing: test code without executing it

#32

Post by motorest »

vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

> That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently (...)

The problem with your assertion is that it fails to understand that today's software, where every single line of code was typed in by real flesh-and-bone humans, already fails to have adequate test coverages, let alone be validated.

The main problem with output from LLMs is that they were trained with the code written by humans, and thus they accurately reflect the quality of the code that's found in the wild. Consequently, your line of reasoning actually criticizes LLMs for outputing the same unreliable code that people write.

Counterintuitively, LLMs end up generating a better output because at least they are designed to simplify the task of automatically generating tests.

Re: Semantic unit testing: test code without executing it

#33

Post by carlmr »

Garlef wrote: post #20
RainyDayTmrw wrote: post #15

I'm skeptical. Most of us maintaining medium sized codebases or larger are constantly fighting nondeterminism in the form of flaky tests. I can't imagine choosing a design that starts with nondeterminism baked in. And if you're really dead-set on paying nondeterminism to get more coverage, property-based testing has existed for a long time and has a comparatively solid track record.

Hm... I think you have a good point. Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes? I think there are also other problems (inlining a verbal description makes the codebase verbose, writing a precise, non-ambiguous verbal description might be more work than writing unit tests)

>Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes?

That would be good anyway to keep the costs reasonable.

Re: Semantic unit testing: test code without executing it

#35

Post by spiddy »

dragonwriter wrote: post #13

This is more of "LLM code review" than any kind of testing, and calling it "testing" is just badly misleading.

this. Let’s not confuse meanings. There are multiple ways to improve quality of code. Testing is one, code review is another. this belongs to the latter

Re: Semantic unit testing: test code without executing it

#36

Post by PeterStuer »

vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

If you are working with natural language, it is by definition 'fuzzy' unless you reduce it to simple templates. So to evaluate whether an output is a semantically e.g. a reasonable answer to an input where non-templated natural verbalization is needed, you need something that 'tests' the output, and that is not going to be purely 'logical'.

Will that test be perfect? No. But what is the alternative?

Re: Semantic unit testing: test code without executing it

#37

Post by InkCanon »

vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

It's a common idea, all the way back to Hoare logic. There was a time when people believed in the future, people would write specifications instead of code.

The problem with it takes several times more effort to verify code than to write it. This makes intuitive sense if you consider that the search space for the properties of code is much larger than the code for space. Rice theorem's states that all non trivial semantic properties of a program are undeniable.

Re: Semantic unit testing: test code without executing it

#38

Post by vouwfietsman »

PeterStuer wrote: post #36
vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

If you are working with natural language, it is by definition 'fuzzy' unless you reduce it to simple templates. So to evaluate whether an output is a semantically e.g. a reasonable answer to an input where non-templated natural verbalization is needed, you need something that 'tests' the output, and that is not going to be purely 'logical'. Will that test be perfect? No. But what is the alternative?

Are you referring to the process of requirement engineering? Because although I agree its a fuzzy natural language interface, behind the interface should be (heavy should) a rigorously defined & designed system, where fuzzyness is eliminated. The LLMs need to work primarily with the rigorous definition, not the fuzzyness.

Re: Semantic unit testing: test code without executing it

#40

Post by vouwfietsman »

motorest wrote: post #32
vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

> That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently (...) The problem with your assertion is that it fails to understand that today's software, where every single line of code was typed in by real flesh-and-bone humans, already fails to have adequate test cover…

Right but by your reasoning it would make sense to use LLMs only to augment an incomplete but rigorous testing process, or to otherwise elevate below average code.

My issue is not necessarily with the quality of the code, but rather with the intention of the code, which is much more important: a good design without tests is more durable than a bad design with tests.

Post reply on HN