Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#1

Post by todsacerdoti » Thu, Apr 10, 2025, 8:10 PM UTC

Cargo-mutants:zombie: Inject bugs and see if your tests catch them
github.com

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#2

Post by matthewfcarlson » Sun, Apr 13, 2025, 3:55 PM UTC

I wonder if there’s something similar to run on a c codebase.

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#3

Post by zhaoweny » Sun, Apr 13, 2025, 4:35 PM UTC

matthewfcarlson wrote: post #2

I wonder if there’s something similar to run on a c codebase.

There's mull[1], which is based on LLVM toolchain and can accept any language that transforms into LLVM bytecode. SQLite[2] also does mutation testing, by compile and mutate generated assembly code.

[1]: https://mull.readthedocs.io/en/0.26.0/ [2]: https://sqlite.org/testing.html#mutation_testing

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#4

Post by BadBadJellyBean » Sun, Apr 13, 2025, 4:36 PM UTC

matthewfcarlson wrote: post #2

I wonder if there’s something similar to run on a c codebase.

There are a few. See: https://github.com/theofidry/awesome-mutation-testing

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#5

Post by rao-v » Sun, Apr 13, 2025, 4:52 PM UTC

This seems like a near perfect use of coding LLMs and a useful way to implement reinforcement learning.

“Add a major bug to this file that is not covered by existing tests” vs “Find the bug in this file” vs “write a sensible test in this file that protects against this type of bug”

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#6

Post by J_Shelby_J » Sun, Apr 13, 2025, 5:28 PM UTC

Neat.

Is this to be used in addition to the tools mentioned in this talk: https://youtube.com/watch?v=qfknfCsICUM

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#7

Post by ramon156 » Sun, Apr 13, 2025, 7:02 PM UTC

Need to replace the :zombie: with a

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#8

Post by infogulch » Sun, Apr 13, 2025, 7:05 PM UTC

rao-v wrote: post #5

This seems like a near perfect use of coding LLMs and a useful way to implement reinforcement learning. “Add a major bug to this file that is not covered by existing tests” vs “Find the bug in this file” vs “write a sensible test in this file that protects against this type of bug”

Somehow that reminds me of how diffusion models are trained.

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#9

Post by stblack » Sun, Apr 13, 2025, 7:34 PM UTC

At RustConf 2024 in Montreal, Cargo-mutants' creator Martin Pool's presentation was excellent. One of the best sessions of the conference.

https://www.youtube.com/watch?v=PjDHe-PkOy8

Re: Cargo-mutants:zombie: Inject bugs and see if your tests catch them

#10

Post by kilroy123 » Sun, Apr 13, 2025, 9:12 PM UTC

Really cool! I wish there was such a thing for JavaScript.

I say this as a so-so software engineer. I badly wish there was more emphasis on increasing software quality. There is so much the industry could do to radically improve quality, such as tools like this.

I know the incentives just aren't there, but still.