Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Semantic unit testing: test code without executing it

alexmolas.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Semantic unit testing: test code without executing it

#51

Post by lgiordano_notte »

vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

LLM-based coding only really works when wrapped in structured prompts, constrained outputs, external checks etc. The systems that work well aren’t just 'LLM take the wheel' architecture, they’re carefully engineered pipelines. Most success stories are more about that scaffolding than the model itself.

Re: Semantic unit testing: test code without executing it

#53

Post by PeterStuer »

vouwfietsman wrote: post #38
PeterStuer wrote: post #36

Earlier quoted context omitted.

If you are working with natural language, it is by definition 'fuzzy' unless you reduce it to simple templates. So to evaluate whether an output is a semantically e.g. a reasonable answer to an input where non-templated natural verbalization is needed, you need something that 'tests' the output, and that is not going to be purely 'logical'. Will that test be perfect? No. But what is the alternative?

Are you referring to the process of requirement engineering? Because although I agree its a fuzzy natural language interface, behind the interface should be (heavy should) a rigorously defined & designed system, where fuzzyness is eliminated. The LLMs need to work primarily with the rigorous definition, not the fuzzyness.

It depends on the use case. e.g. Music generation like Suno. How do you rigorously and logically check the output? Or an automated copy-writing service?

The tests should match the rigidity of the case. A mismatch in modality will lead to bad outcomes.

Re: Semantic unit testing: test code without executing it

#55

Post by vouwfietsman »

motorest wrote: post #47
vouwfietsman wrote: post #40

Earlier quoted context omitted.

Right but by your reasoning it would make sense to use LLMs only to augment an incomplete but rigorous testing process, or to otherwise elevate below average code. My issue is not necessarily with the quality of the code, but rather with the intention of the code, which is much more important: a good design without tests is more durable than a bad design with tests.

> Right but by your reasoning it would make sense to use LLMs only to augment an incomplete but rigorous testing process, or to otherwise elevate below average code. No. It makes sense to use LLMs to generate tests. Even if their output matches the worst output the average human can write by hand, having any coverage whatsoever already raises the bar from where the average human output is. > My issue is not necessari…

> No. It makes sense to use LLMs to generate tests. Even if their output matches the worst output the average human can write by hand, having any coverage whatsoever already raises the bar from where the average human output is.

Although this is true, it disregards the fact that prompting for tests takes time which may also be spent writing tests, and its not clear if poor quality tests are free, in the sense that further development may cause these tests to fail for the wrong reasons, causing time spent debugging. This is why I used the word "augment": these tests are clearly not the same quality as manual tests, and should be considered separately from manual tests. In other words, they may serve to elevate below average code or augment manual tests, but not more than that. Again, I'm not saying it makes no sense to do this.

> That's not the LLM's responsibility. Humans specify what they want and LLMs fill in the blanks. If today's LLMs output bad results, that's a reflection of the prompts. Garbage in, garbage out.

This is unlikely to be true, for a couple reasons: 1. Ambiguity makes it impossible to define "garbage", see prompt engineering. In fact, all human natural language output is garbage in the context of programming. 2. As the LLM fills in blanks, it must do so respecting the intention of the code, otherwise the intention of the code erodes, and its design is lost. 3. This would imply that LLMs have reached their peak and only improve by requiring less prompting by a user, this is simply not true as it is trivial to currently find problems an LLM cannot solve, regardless of the amount of prompting.

Re: Semantic unit testing: test code without executing it

#56

Post by CivBase »

lgiordano_notte wrote: post #51
vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

LLM-based coding only really works when wrapped in structured prompts, constrained outputs, external checks etc. The systems that work well aren’t just 'LLM take the wheel' architecture, they’re carefully engineered pipelines. Most success stories are more about that scaffolding than the model itself.

Does anyone provide a good breakdown of how much time/cost goes into the scaffolding vs how much is saved from not writing the code itself?

Re: Semantic unit testing: test code without executing it

#57

Post by vouwfietsman »

PeterStuer wrote: post #53
vouwfietsman wrote: post #38

Earlier quoted context omitted.

Are you referring to the process of requirement engineering? Because although I agree its a fuzzy natural language interface, behind the interface should be (heavy should) a rigorously defined & designed system, where fuzzyness is eliminated. The LLMs need to work primarily with the rigorous definition, not the fuzzyness.

It depends on the use case. e.g. Music generation like Suno. How do you rigorously and logically check the output? Or an automated copy-writing service? The tests should match the rigidity of the case. A mismatch in modality will lead to bad outcomes.

Aha! Like that. Yes that's interesting, the only other alternative would be manual classification of novel data, so extremely labour intensive. If an LLM is able to do the same classification automatically it opens up use cases that are otherwise indeed impossible.

Re: Semantic unit testing: test code without executing it

#58

Post by bluGill »

lgiordano_notte wrote: post #49

Treating docstrings as the spec and asking an LLM to flag mismatches feels promising in theory but personally I'd b wary of overfitting to underspecified docs. Might be useful as a lint-like signal, but hard to see it replacing real tests just yet.

if that is the only testing you do I agree. However to test that the code works as the docs say is valuable as well. The code often will do more, but it needs to do at least what the docs say.

Re: Semantic unit testing: test code without executing it

#59

Post by motorest »

vouwfietsman wrote: post #55
motorest wrote: post #47

Earlier quoted context omitted.

> Right but by your reasoning it would make sense to use LLMs only to augment an incomplete but rigorous testing process, or to otherwise elevate below average code. No. It makes sense to use LLMs to generate tests. Even if their output matches the worst output the average human can write by hand, having any coverage whatsoever already raises the bar from where the average human output is. > My issue is not necessari…

> No. It makes sense to use LLMs to generate tests. Even if their output matches the worst output the average human can write by hand, having any coverage whatsoever already raises the bar from where the average human output is. Although this is true, it disregards the fact that prompting for tests takes time which may also be spent writing tests, and its not clear if poor quality tests are free, in the sense that fu…

> Although this is true, it disregards the fact that prompting for tests takes time which may also be spent writing tests (...)

No, not today at least. Some services like Copilot provide plugins that implement actions to automatically generate unit tests. This means that the unit test coverage you're describing is a right-click away.

https://code.visualstudio.com/docs/copilot/copilot-smart-act...

> (...).and its not clear if poor quality tests are free, in the sense that further development may cause these tests to fail for the wrong reasons, causing time spent debugging.

That's not how automated tests work. If you have a green test that turns red when you touch some part of the code, this is the test working as expected, because your code change just introduced unexpected changes that violated an invariant.

Also, today's LLMs are able to recreate all your unit tests from scratch.

> This is unlikely to be true, for a couple reasons: 1. Ambiguity makes it impossible to define "garbage", see prompt engineering.

"Ambiguity" is garbage in this context.

> . 2. As the LLM fills in blanks, it must do so respecting the intention of the code, otherwise the intention of the code erodes, and its design is lost.

That's the responsibility of the developer, not the LLM. Garbage in, garbage out.

> . 3. This would imply that LLMs have reached their peak and only improve by requiring less prompting by a user, this is simply not true as it is trivial to currently find problems an LLM cannot solve, regardless of the amount of prompting.

I don't think that point is relevant. The goal of a developer is still to meet the definition of done, not to tie their hands around their back and expect working code to just fall on their lap. Currently the main approach to vibe coding is to set the architecture, and lean on the LLM to progressively go from high level to low level details. Speaking from personal experience in vibecoding, LLMs are quite capable of delivering fully working apps with a single, detailed prompt. However, you get far more satisfactory results (i.e., the app reflects the same errors in judgement you'd make) if you just draft a skeleton and progressively fill in the blanks.

Re: Semantic unit testing: test code without executing it

#60

Post by bluGill »

0xDEAFBEAD wrote: post #22
motorest wrote: post #10

Earlier quoted context omitted.

> But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this. I firmly believe that the group of people who laud dynamically typed languages as efficient time-savers, that help shed drudge work involving typing, is tightly correlated with the group of people who fail to establish any form of quality assurance or testing, often using the same arguments t…

The question I find interesting is whether type systems are an efficient way to buy reliability relative to other ways to purchase reliability, such as writing tests, doing code review, or enforcing immutability. Of course, some programmers just don't care about purchasing reliability. Those are the ones who eschew type systems, and tests, and produce unreliable software, about like you'd expect. But for my purposes,…

I find they are valuable. When you have a small program - 10k lines of code you don't really need them. However when you are at more than 10 million lines of code types find a lot of little errors that writing the correct test for would be hard.

Most dynamically typed languages (all that I have worked with) cannot catch that you misspelled a function name until that function is called. If that misspelled function is in an error path it would be very easy to never test it until a customer hit the crash. Just having your function names as a strong type that is checked by static analysis (need not be a compiler though that is what everything uses) is a big win. Checking the other arguments as well is similarly helpful.

Post reply on HN