Automated Test-Case Reduction
cs.cornell.edu
Automated Test-Case Reduction
1–10 of 30 posts
Re: Automated Test-Case Reduction
#2I find it odd that test-case reduction is billed as a “research skill”. To me, this is a basic debugging skill for anyone who works with code.
Maybe it’s more critical when you’re dealing with “research-quality” code. This post seems to ignore the underlying problem: it’s rare to see research code with any tests.
Re: Automated Test-Case Reduction
#3I find it odd that test-case reduction is billed as a “research skill”. To me, this is a basic debugging skill for anyone who works with code. Maybe it’s more critical when you’re dealing with “research-quality” code. This post seems to ignore the underlying problem: it’s rare to see research code with any tests.
As the article says, right at the top: "The plan is to demonstrate techniques that “everyone knows” because everyone, in fact, does not already know them."
Re: Automated Test-Case Reduction
#4I find it odd that test-case reduction is billed as a “research skill”. To me, this is a basic debugging skill for anyone who works with code. Maybe it’s more critical when you’re dealing with “research-quality” code. This post seems to ignore the underlying problem: it’s rare to see research code with any tests.
As the article says, right at the top: "The plan is to demonstrate techniques that “everyone knows” because everyone, in fact, does not already know them."
This is actually something I find hard to resist in myself as an aging software engineer on HN. When I was young, articles would describe “obvious” things in excruciating detail, and I’d be grateful, because they were often things I didn’t know.
Now when I see articles about things I find obvious on the front page — particularly if they’re topics I’ve seen hit the front page before (which, mind you, means “any time in the last 15 years or so”) — I have to resist an annoyed gut reaction. The fact that knowledge and skills seem obvious and boring once internalized is a kind of terrible quirk of the human condition.
Re: Automated Test-Case Reduction
#5Earlier quoted context omitted.
As the article says, right at the top: "The plan is to demonstrate techniques that “everyone knows” because everyone, in fact, does not already know them."
This is actually something I find hard to resist in myself as an aging software engineer on HN. When I was young, articles would describe “obvious” things in excruciating detail, and I’d be grateful, because they were often things I didn’t know. Now when I see articles about things I find obvious on the front page — particularly if they’re topics I’ve seen hit the front page before (which, mind you, means “any time i…
For whatever it's worth, I enjoy explaining things to others so each time a topic recurs I can do that where it seems appropriate. And hey I can pass on things I didn't know about and then learned here, for example that's why I tell people about WUFFS, can you do all the bounds checking at compile time? Sure, for a price. And that price (generality) might be easily affordable in which case using Rust, never mind C++ is very silly because WUFFS was offering you better performance and total safety if you can do without generality.
Re: Automated Test-Case Reduction
#6In addition to c-reduce and shrinkray there's also cvise [1] for reducing tests. It's great.
Also, if you happen to be reducing an LLVM case you can use llvm-reduce [2].
Re: Automated Test-Case Reduction
#7Earlier quoted context omitted.
As the article says, right at the top: "The plan is to demonstrate techniques that “everyone knows” because everyone, in fact, does not already know them."
This is actually something I find hard to resist in myself as an aging software engineer on HN. When I was young, articles would describe “obvious” things in excruciating detail, and I’d be grateful, because they were often things I didn’t know. Now when I see articles about things I find obvious on the front page — particularly if they’re topics I’ve seen hit the front page before (which, mind you, means “any time i…
My issue isn't with articles describing "obvious" things, it's with articles acting like this obvious thing is an amazing new discovery. This is especially funny when you see it in articles about relational database stuff, since most of the fundamentals have been discovered since the 70's.
Re: Automated Test-Case Reduction
#8Very interesting. Previously I've seen two similar ideas:
1. Reduce the test data by property-based tests' auto shrinking a là QuickCheck [1]
2. Reduce the change set to a codebase, a là delta debugging [2]
The reducer is able to do both at the same time in a pretty generic way, with some restrictions on the expressivity. Seems to be a good fit for end-to-end tests.
Re: Automated Test-Case Reduction
#9I used to do something like this all the time with C/C++ compiler tests. I tried lots of fancy tools and stuff, but I kept going back to: expand all macros and retokenize to one token per line (I made a custom build of the preprocessor that had this built in). Then, have a shell script randomly remove lines, and use another script to check that the resulting test case behaves consistently with the failure. It would run for a few hours (or days, for boost problems), then usually you'd get a really minimal testcase that shows the problem. Often I would use this to find regressions. Just have the shell script check one is good, the other has the problem. The resulting output would then usually point exactly at the regressed feature, and make an amazing unit test.
Before leaving compiler team, I wanted to find a way to do this one the AST level (so parens always go in pairs, etc), but that could also complect with the bug.
I wonder if LLMs could accelerate this by more intelligently removing stuff first, iteratively?
Re: Automated Test-Case Reduction
#10I find it odd that test-case reduction is billed as a “research skill”. To me, this is a basic debugging skill for anyone who works with code. Maybe it’s more critical when you’re dealing with “research-quality” code. This post seems to ignore the underlying problem: it’s rare to see research code with any tests.
I don't understand your point. Debugging is research almost by definition, no?