Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Semantic unit testing: test code without executing it

alexmolas.com

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Semantic unit testing: test code without executing it

#21

Post by 0xDEAFBEAD »

jonathanlydall wrote: post #6

If you’re stuck with dynamically typed languages, then tests like this can make a lot of sense. On statically typed languages this happens for free at compile time. I’ve often heard proponents of dynamically typed languages say how all the typing and boiler plate required by statically typed languages feels like such a waste of time, and on a small enough system maybe they are right. But on any significant sized code…

Dan Luu looked at the literature and concluded that the evidence for the benefit of types is underwhelming:

https://danluu.com/empirical-pl/

>But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this.

OK, but if the alternative to tests is spending more time on a reliability method (type annotations) which buys you less reliability compared to writing tests... it's hardly a win.

It fundamentally seems to me that there are plenty of bugs that types can simply never catch. For example, if I have a "divide" function and I accidentally swap the numerator and divisor arguments, I can't think of any realistic type system which will help me. Other methods for achieving reliability, like writing tests or doing code review, don't seem to have the same limitations.

Re: Semantic unit testing: test code without executing it

#22

Post by 0xDEAFBEAD »

motorest wrote: post #10
jonathanlydall wrote: post #6

If you’re stuck with dynamically typed languages, then tests like this can make a lot of sense. On statically typed languages this happens for free at compile time. I’ve often heard proponents of dynamically typed languages say how all the typing and boiler plate required by statically typed languages feels like such a waste of time, and on a small enough system maybe they are right. But on any significant sized code…

> But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this. I firmly believe that the group of people who laud dynamically typed languages as efficient time-savers, that help shed drudge work involving typing, is tightly correlated with the group of people who fail to establish any form of quality assurance or testing, often using the same arguments t…

The question I find interesting is whether type systems are an efficient way to buy reliability relative to other ways to purchase reliability, such as writing tests, doing code review, or enforcing immutability.

Of course, some programmers just don't care about purchasing reliability. Those are the ones who eschew type systems, and tests, and produce unreliable software, about like you'd expect. But for my purposes, this is besides the point.

Re: Semantic unit testing: test code without executing it

#23

Post by globular-toast »

motorest wrote: post #10
jonathanlydall wrote: post #6

If you’re stuck with dynamically typed languages, then tests like this can make a lot of sense. On statically typed languages this happens for free at compile time. I’ve often heard proponents of dynamically typed languages say how all the typing and boiler plate required by statically typed languages feels like such a waste of time, and on a small enough system maybe they are right. But on any significant sized code…

> But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this. I firmly believe that the group of people who laud dynamically typed languages as efficient time-savers, that help shed drudge work involving typing, is tightly correlated with the group of people who fail to establish any form of quality assurance or testing, often using the same arguments t…

Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness. I could just as easily say static people don't write tests because they think the type system is enough. A type system is laughably bad at asserting correct behaviour.

Personally I do use type hinting and mypy for much of my Python code. But I'll most certainly omit it for throwaway scripts and trivial stuff. I'm still not convinced it's really worth the effort, though. I've had a few occasions where the type checker has caught something important, but most of the time it's an autist trap where you spend ages making it correct "just because".

Re: Semantic unit testing: test code without executing it

#24

Post by anself »

dragonwriter wrote: post #13

This is more of "LLM code review" than any kind of testing, and calling it "testing" is just badly misleading.

Agree, it's not testing. The problem is here: "In a typical testing workflow, you write some basic tests to check the core functionality. When a bug inevitably shows up—usually after deployment—you go back and add more tests to cover it. This process is reactive, time-consuming, and frankly, a bit tedious."

This is exactly the problem that TDD solves. One of the most compelling reasons for test-first is because "Running the code in your head" does not actually work well in practice, leading to the above-cited issues. This is just another variant of "Running the code in your head" except an LLM is doing it. Strong TDD practices (don't write any code without a test to support it) will close those gaps. It may feel tedious at first but the safety it creates will leave you never wanting to go back.

Where this could be safe and useful: Find gaps in the test-set. Places where the code was never written because there wasn't a test to drive it out. This is one of the hardest parts of TDD, and where LLMs could really help.

Re: Semantic unit testing: test code without executing it

#25

Post by cerpins »

It sounds like it might be a good use case for testing documentation - verifying whether what documentation describes is actually in accordance with the code, and then you can act on it. With that in mind, it's also probably pointless to re-run if relevant code or documentation hasn't changed.

Re: Semantic unit testing: test code without executing it

#26

Post by vouwfietsman »

Maybe someone can help me out here:

I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation must be on par with human authored code + review. Because of this, the validation cannot be done by an LLM, as it would just compound the problem.

Unless we get a drastic change in the level of error detection and self-validation that can be done by an LLM, this remains a problem for the foreseeable future.

How is it then that people build tooling where the LLM validates the code they write? Or claim 2x speedups for code written by LLMs? Is there some kind of false positive/negative tradeoff I'm missing that allows people to extract robust software from an inherently not-robust generation process?

I'm not talking about search and documentation, where I'm already seeing a lot of benefit from LLMs today, because between the LLM output and the code is me, sanity checking and filtering everything. What I'm asking about is the: "LLM take the wheel!" type engineering.

Re: Semantic unit testing: test code without executing it

#27

Post by darawk »

vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

This particular person seems to be using LLMs for code review, not generation. I agree that the problem is compounded if you use an LLM (esp. the same model) on both sides. However, it seems reasonable and useful to use it as an adjunct to other forms of testing, though not necessarily a replacement for them. Though again, the degree to which it can be a replacement is a function of the level of the technology, and it is currently at the level where it can probably replace some traditional testing methods, though it's hard to know which, ex-ante.

edit: of course, maybe that means we need a meta-suite, that uses a different LLM to tell you which tests you should write yourself and which tests you can safely leave to LLM review.

Re: Semantic unit testing: test code without executing it

#28

Post by sigtstp »

I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave.

"Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but that is an assertion which is contested [1].

Coming back to testing, this implicitly relies on the strong assumption of the LLM correctly associating the code (syntax) with assertions of properties under execution (semantic properties). This is a very risky assumption considering, once again, these things are stochastic in nature and cannot even guarantee syntactical correctness, let alone semantic. Being generous with the former, there is a track record of the latter often failing and producing subtle bugs [2][3][4][5]. Not to mention the observed effect of LLMs often being biased to "agree" with the premise presented to them.

It also kind of misses the point of testing, which is the engineering (not automation) task of reasoning about code and doing QC (even if said tests are later run automatically, I'm talking about their conception). I feel it's a dangerous, albeit tempting, decision to relegate that to an LLM. Fuzzing, sure. But not assertions about program behavior.

[1] A Primer in BERTology: What we know about how BERT works https://arxiv.org/abs/2002.12327 (Layers encode a mix of syntactic and semantic aspects of natural language, and it's problem-specific.)

[2] Large Language Models of Code Fail at Completing Code with Potential Bugs https://arxiv.org/abs/2306.03438

[3] SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? https://arxiv.org/abs/2502.12115 (best models unable to solve the majority of coding problems)

[4] Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT https://arxiv.org/abs/2304.10778

[5] Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions https://arxiv.org/abs/2308.02312v4

EDIT: Added references

Re: Semantic unit testing: test code without executing it

#29

Post by motorest »

globular-toast wrote: post #23
motorest wrote: post #10

Earlier quoted context omitted.

> But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this. I firmly believe that the group of people who laud dynamically typed languages as efficient time-savers, that help shed drudge work involving typing, is tightly correlated with the group of people who fail to establish any form of quality assurance or testing, often using the same arguments t…

Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness. I could just as easily say static people don't write tests because they think the type system is enough. A type system is laughably bad at asserting correct behaviour. Personally I do use type hinting and mypy for much of my Python code. But I'll most certainly omit it fo…

> Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness.

Tests don't assert correctness. At best they verify specific invariants.

Statically typed languages lean on the compiler to automatically verify some classes of invariants (i.e., can I call this method in this object?)

With dynamically typed languages, you cannot lean on the compiler to verify these invariants. Developers must fill in this void by writing their own tests.

It's true that they "need" to do it to avoid some classes of runtime errors that are only possible in dynamically typed languages. But that's not the point. The point is that those who complan that statically typed languages are too cumbersome because they require boilerplate code for things type compile-time type checking are also correlated with the set of developers who fail to invest any time adding or maintaining automated test suites, because of the same reasons.

> I could just as easily say static people don't write tests because they think the type system is enough. A type system is laughably bad at asserting correct behaviour.

No, you can't. Developers who use statically typed languages don't even think of type checking as a concern, let alone a quality assurance issue.

Post reply on HN