Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Semantic unit testing: test code without executing it

alexmolas.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Semantic unit testing: test code without executing it

#61

Post by bluGill »

motorest wrote: post #29
globular-toast wrote: post #23

Earlier quoted context omitted.

Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness. I could just as easily say static people don't write tests because they think the type system is enough. A type system is laughably bad at asserting correct behaviour. Personally I do use type hinting and mypy for much of my Python code. But I'll most certainly omit it fo…

> Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness. Tests don't assert correctness. At best they verify specific invariants. Statically typed languages lean on the compiler to automatically verify some classes of invariants (i.e., can I call this method in this object?) With dynamically typed languages, you cannot lean o…

> Tests don't assert correctness. At best they verify specific invariants.

Pedantically correct, but in practice those are close enough to the same thing.

Even a formal proof cannot assert correctness - requirements are often wrong. However in practice requirements are close enough to correct that we can call a formal proof also close enough.

Re: Semantic unit testing: test code without executing it

#62

Post by bluGill »

motorest wrote: post #11
yuliyp wrote: post #7

Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebases? What's the false positive rate? False negative? As is it's neat that they wrote some code to generate some prompts for an LLM but there's no idea if it actually works.

> Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebases? What's the false positive rate? False negative? I would also add the concern on whether the tests are actually deterministic. The premise is also dubious, as docstring comments typically hold only very high-level descriptions of the implementation and oft…

Documentation should not be telling your how it is implemented. It should tell you how and why to use the function. Users who care about how it is implemented should be reading the code not the comments. Users who need to find/use a helper and get on with their feature shouldn't.

Re: Semantic unit testing: test code without executing it

#63

Post by lgiordano_notte »

CivBase wrote: post #56
lgiordano_notte wrote: post #51

Earlier quoted context omitted.

LLM-based coding only really works when wrapped in structured prompts, constrained outputs, external checks etc. The systems that work well aren’t just 'LLM take the wheel' architecture, they’re carefully engineered pipelines. Most success stories are more about that scaffolding than the model itself.

Does anyone provide a good breakdown of how much time/cost goes into the scaffolding vs how much is saved from not writing the code itself?

A breakdown would be interesting. I can’t give you hard numbers, but in our case scaffolding was most of the work. Getting the model to act reliably meant building structured abstractions, retries, output validation, context tracking, etc. Once that’s in place you start saving time per task, but there’s a cost up front.

Re: Semantic unit testing: test code without executing it

#64

Post by lgiordano_notte »

bluGill wrote: post #58
lgiordano_notte wrote: post #49

Treating docstrings as the spec and asking an LLM to flag mismatches feels promising in theory but personally I'd b wary of overfitting to underspecified docs. Might be useful as a lint-like signal, but hard to see it replacing real tests just yet.

if that is the only testing you do I agree. However to test that the code works as the docs say is valuable as well. The code often will do more, but it needs to do at least what the docs say.

Agreed. Catching mismatches between doc and implementation is still valuable, just wouldn’t want people to rely on it as a safety net when the docs themselves might be inaccurate/incomplete. As a complement to traditional tests though seems like a solid addition.

Re: Semantic unit testing: test code without executing it

#65

Post by vouwfietsman »

motorest wrote: post #59
vouwfietsman wrote: post #55

Earlier quoted context omitted.

> No. It makes sense to use LLMs to generate tests. Even if their output matches the worst output the average human can write by hand, having any coverage whatsoever already raises the bar from where the average human output is. Although this is true, it disregards the fact that prompting for tests takes time which may also be spent writing tests, and its not clear if poor quality tests are free, in the sense that fu…

> Although this is true, it disregards the fact that prompting for tests takes time which may also be spent writing tests (...) No, not today at least. Some services like Copilot provide plugins that implement actions to automatically generate unit tests. This means that the unit test coverage you're describing is a right-click away. https://code.visualstudio.com/docs/copilot/copilot-smart-act... > (...).and its not…

> That's not how automated tests work > today's LLMs are able to recreate all your unit tests from scratch. > That's the responsibility of the developer > LLMs are quite capable of delivering fully working apps with a single, detailed prompt

You seem to be very resolute in positing generalizations, I think those are rarely true. I don't see a lot of benefit coming out of a discussion like this. Try reading my replies as if you agree with them, it will help you better understand my point of view, which will make your criticism more targeted, so you can avoid generalizations.

Re: Semantic unit testing: test code without executing it

#66

Post by gavmor »

If you don't try static typing, first, I feel like you're leaving money on the table... on your way to burn a pile of money.

Right? If you're looking to reduce bugs and errors... this is like putting a jetpack on a window-washer without even considering a carabiner harness.

Re: Semantic unit testing: test code without executing it

#67

Post by RainyDayTmrw »

Davidbrcz wrote: post #45
RainyDayTmrw wrote: post #15

I'm skeptical. Most of us maintaining medium sized codebases or larger are constantly fighting nondeterminism in the form of flaky tests. I can't imagine choosing a design that starts with nondeterminism baked in. And if you're really dead-set on paying nondeterminism to get more coverage, property-based testing has existed for a long time and has a comparatively solid track record.

Many good and prolific approaches are non deterministic such as fuzzing or property-based testing,

Both fuzzing and property-based testing, under certain assumptions, produce only false negatives, never false positives. That is a very desirable trait, because then you can at least confidently fail CI on a positive, because either your code is wrong or your test is wrong, and you've got something to fix no matter what.

Re: Semantic unit testing: test code without executing it

#68

Post by 0xDEAFBEAD »

bluGill wrote: post #60
0xDEAFBEAD wrote: post #22

Earlier quoted context omitted.

The question I find interesting is whether type systems are an efficient way to buy reliability relative to other ways to purchase reliability, such as writing tests, doing code review, or enforcing immutability. Of course, some programmers just don't care about purchasing reliability. Those are the ones who eschew type systems, and tests, and produce unreliable software, about like you'd expect. But for my purposes,…

I find they are valuable. When you have a small program - 10k lines of code you don't really need them. However when you are at more than 10 million lines of code types find a lot of little errors that writing the correct test for would be hard. Most dynamically typed languages (all that I have worked with) cannot catch that you misspelled a function name until that function is called. If that misspelled function is…

Thanks for the reply!

Technically, I would say that you should just write a test for the error path.

I also tend to think that finding bugs at compile time is not actually that much better than finding them at runtime. If you have good monitoring (which you should have), and a customer triggers the bug at runtime, you can fix the bug then. A customer triggering a rare bug once isn't the end of the world.

My sense is that people are very keen to find bugs at compile time rather than run time, but I haven't been generally able to figure out why this matters, unless you're working on some sort of real-time embedded system. What matters to me is the probability of spotting the bug, not when it is spotted.

I suppose perhaps data corruption could be a concern with runtime errors? I would argue you should work to be robust against that anyways, by using atomic transactions as necessary.

I'm most interested in subtler bugs which don't trigger exceptions at runtime. Those are the sort of bugs which will get executed over and over in production if they aren't prevented. It seems to me that static analysis doesn't do much to prevent these bugs, but tests can be really helpful.

I will grant that implicit type conversions can lead to subtle runtime bugs. I see that as a separate issue however. I've worked most with Python, which doesn't do static analysis, but also enforces types at runtime. That seems fairly ideal to me.

Interested to hear your perspective!

Re: Semantic unit testing: test code without executing it

#69

Post by 0xDEAFBEAD »

Smaug123 wrote: post #48
0xDEAFBEAD wrote: post #21

Earlier quoted context omitted.

Dan Luu looked at the literature and concluded that the evidence for the benefit of types is underwhelming: https://danluu.com/empirical-pl/ >But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this. OK, but if the alternative to tests is spending more time on a reliability method (type annotations) which buys you less reliability compared to writing…

> swap the numerator and divisor Even Rust can express this; you don't need to get fancy. Morally speaking, division takes a Num and a std::num::NonZero .

OK, I guess you got me for the particular case of division. Have your upvote.

Re: Semantic unit testing: test code without executing it

#70

Post by globular-toast »

motorest wrote: post #29
globular-toast wrote: post #23

Earlier quoted context omitted.

Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness. I could just as easily say static people don't write tests because they think the type system is enough. A type system is laughably bad at asserting correct behaviour. Personally I do use type hinting and mypy for much of my Python code. But I'll most certainly omit it fo…

> Rubbish, in my experience. People who understand dynamic languages know they need to write tests because it's the only thing asserting correctness. Tests don't assert correctness. At best they verify specific invariants. Statically typed languages lean on the compiler to automatically verify some classes of invariants (i.e., can I call this method in this object?) With dynamically typed languages, you cannot lean o…

Tests assert correctness of behaviour, which is the only thing that really matters. A type system can't do that. Spending time getting your types right just to please the type checker does catch some bugs sometimes, but I'm not convinced it's truly worth it given that bugs usually happen at a much higher level anyway which will lead to rewriting/throwing away the lower level code anyway.

It's a completely different tradeoff if those types are also being used to compile the program into native code. No C programmer complains about types because at least we're getting native code with zero overhead. It's a tradeoff that's worth making, sometimes. But there are so many cases where speed does not matter, and there it's much harder to make the case.

As always, it's not really about static vs dynamic, strong vs weak, red vs blue, left vs right etc. All have their virtues. The axis that really matters is extremity. Extreme adherence to anything is rarely a good thing.

Post reply on HN