Semantic unit testing: test code without executing it
1–10 of 73 posts
Re: Semantic unit testing: test code without executing it
#2Much better solution: don't write useless docstrings.
Re: Semantic unit testing: test code without executing it
#3This seems to be your site @op.. your CSS needs attention. On a narrower screen (ie. portrait) the text is enormous, and worse, zooming out shrinks the quantity of words (increases the font-size).. which is the surely the opposite of expected? It's basically unusable.
Your CSS seems to assume all portrait screens (whether 80" or 3") deserve the same treatment.
Re: Semantic unit testing: test code without executing it
#4This is cool! I think that, in general, generating test cases “offline” using an LLM and then running them using regular unit testing also solves this particular issue.
It also might be more transparent and cheaper.
Re: Semantic unit testing: test code without executing it
#5I was a bit skeptical at first but I think this is a good idea. Although I'm not convinced with the usage of max_depth parameter. In real life you rarely know what type your dependencies are if they are loaded at run time. This is kind of why we explicitly mock our dependencies.
On a side note: I have wondered whether LLM's are particularly good with functional languages. Imagine if your code entirely consisted of just pure functions and no side effects. You pass all parameters required and do not use static methods/variables and no OOP concepts like inheritance. I imagine every program can be converted in such a way, the tradeoff being human readability.
Re: Semantic unit testing: test code without executing it
#6If you’re stuck with dynamically typed languages, then tests like this can make a lot of sense.
On statically typed languages this happens for free at compile time.
I’ve often heard proponents of dynamically typed languages say how all the typing and boiler plate required by statically typed languages feels like such a waste of time, and on a small enough system maybe they are right.
But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this.
They also allow trivial refactoring that people using dynamically typed languages wouldn’t even consider due to the risk being so high.
So keep this all in mind when you next choose your language for a new project.
Re: Semantic unit testing: test code without executing it
#7Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebases? What's the false positive rate? False negative?
As is it's neat that they wrote some code to generate some prompts for an LLM but there's no idea if it actually works.
Re: Semantic unit testing: test code without executing it
#8If you’re stuck with dynamically typed languages, then tests like this can make a lot of sense. On statically typed languages this happens for free at compile time. I’ve often heard proponents of dynamically typed languages say how all the typing and boiler plate required by statically typed languages feels like such a waste of time, and on a small enough system maybe they are right. But on any significant sized code…
I think at least some people who say this think of Java-esque type systems. And there I agree: it is a boilerplate nightmare.
Re: Semantic unit testing: test code without executing it
#9I wonder if the random component of the LLM makes every test flaky by definition.
Re: Semantic unit testing: test code without executing it
#10If you’re stuck with dynamically typed languages, then tests like this can make a lot of sense. On statically typed languages this happens for free at compile time. I’ve often heard proponents of dynamically typed languages say how all the typing and boiler plate required by statically typed languages feels like such a waste of time, and on a small enough system maybe they are right. But on any significant sized code…
> But on any significant sized code bases, they pay dividends over and over by saving you from having to make tests like this.
I firmly believe that the group of people who laud dynamically typed languages as efficient time-savers, that help shed drudge work involving typing, is tightly correlated with the group of people who fail to establish any form of quality assurance or testing, often using the same arguments to justify their motivation.