Semantic unit testing: test code without executing it • Hacker News Board

Re: Semantic unit testing: test code without executing it

#71

Post by bluGill » Tue, May 06, 2025, 1:05 PM UTC

0xDEAFBEAD wrote: post #68

bluGill wrote: post #60

Earlier quoted context omitted.

I find they are valuable. When you have a small program - 10k lines of code you don't really need them. However when you are at more than 10 million lines of code types find a lot of little errors that writing the correct test for would be hard. Most dynamically typed languages (all that I have worked with) cannot catch that you misspelled a function name until that function is called. If that misspelled function is…

Thanks for the reply! Technically, I would say that you should just write a test for the error path. I also tend to think that finding bugs at compile time is not actually that much better than finding them at runtime. If you have good monitoring (which you should have), and a customer triggers the bug at runtime, you can fix the bug then. A customer triggering a rare bug once isn't the end of the world. My sense is…

I'm guessing you don't work on programs with millions of lines of code. What you say makes perfect sense for programs that are only a few thousand lines wrong. Nothing wrong with small programs, there are a lot of useful programs that only need to be that long. However when you start writing something more complex you need all the help you can get, and suddenly tests become a valueable way to ensure you don't break some working feature when you add the next.

Re: Semantic unit testing: test code without executing it

#72

Post by 0xDEAFBEAD » Tue, May 06, 2025, 1:51 PM UTC

bluGill wrote: post #71

0xDEAFBEAD wrote: post #68

Earlier quoted context omitted.

Thanks for the reply! Technically, I would say that you should just write a test for the error path. I also tend to think that finding bugs at compile time is not actually that much better than finding them at runtime. If you have good monitoring (which you should have), and a customer triggers the bug at runtime, you can fix the bug then. A customer triggering a rare bug once isn't the end of the world. My sense is…

I'm guessing you don't work on programs with millions of lines of code. What you say makes perfect sense for programs that are only a few thousand lines wrong. Nothing wrong with small programs, there are a lot of useful programs that only need to be that long. However when you start writing something more complex you need all the help you can get, and suddenly tests become a valueable way to ensure you don't break s…

I think you might be getting mixed up. I'm the one advocating tests. You've been advocating types :-)

Re: Semantic unit testing: test code without executing it

#73

Post by deterministic » Thu, May 08, 2025, 4:27 AM UTC

InkCanon wrote: post #37

vouwfietsman wrote: post #26

Maybe someone can help me out here: I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation mu…

It's a common idea, all the way back to Hoare logic. There was a time when people believed in the future, people would write specifications instead of code. The problem with it takes several times more effort to verify code than to write it. This makes intuitive sense if you consider that the search space for the properties of code is much larger than the code for space. Rice theorem's states that all non trivial sem…

Using a LLM to generate the proofs from a spec and verify it (OK/Error) would make it much faster.