It's always good to write tests with the "The Enterprise Developer from Hell" in mind: https://fsharpforfunandprofit.com/posts/property-based-testi...
This is a great talk. More technical videos should be done from the basics like this.
Property-Based Testing Caught a Security Bug I Never Would Have Found
31–40 of 40 posts
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#32TL;DR: obj[key] with user-controlled key == "__proto__" is a gift that keeps on giving; buy our AI tool that will write subtle vulnerabilities like that which you yourself won’t catch in review but then it will also write some property-based tests that maybe will
It also talks about using PBT and Randomness for some reason. This is clearly just a test value of a non-AI library written by a human.
My take away is “don’t write your own input tests, use a library”. The rest is AI-slip
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#33Earlier quoted context omitted.
The original claim for TDD is your write tests for all your edge cases. It doesn't matter about inputs you didn't consider because they are covered in the edge. If you can only accept inputs from 2-7 (inclusive) you check 1,2,7,8 - if those pass you assume the rest work.
You forgot 0, -1, null, "1".
Since I work in a strongly typed languages the last two will fail to compile and are thus not worth the bother - those who don't have that luxury of course need to test the edge cases that apply to them. The first are maybe, in my experience they are rarely a problem, but we need to go from the abstract to the particular algorithm before we can have a discussion on if they are potentially a problem or not.
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#34TL;DR: obj[key] with user-controlled key == "__proto__" is a gift that keeps on giving; buy our AI tool that will write subtle vulnerabilities like that which you yourself won’t catch in review but then it will also write some property-based tests that maybe will
It also talks about using PBT and Randomness for some reason. This is clearly just a test value of a non-AI library written by a human. My take away is “don’t write your own input tests, use a library”. The rest is AI-slip
Theoretically a good fuzzer could discover this value by itself, but I don’t believe anything like that exists that could run JS code and explore VM-level branches, at least not for JS code that’s even this complex. Otherwise, yes, PBT is less trivial than it seems, though I’m guessing a simple `strings jsc` coupled with general knowledge of special values of other types[1,2] could get you quite far.
[1] https://www.exploringbinary.com/php-hangs-on-numeric-value-2...
[2] https://www.exploringbinary.com/java-hangs-when-converting-2...
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#35TL;DR: obj[key] with user-controlled key == "__proto__" is a gift that keeps on giving; buy our AI tool that will write subtle vulnerabilities like that which you yourself won’t catch in review but then it will also write some property-based tests that maybe will
Didn't react just have basically the same vuln
The code in TFA is, by a hair’s breadth, not actually vulnerable, as long as the type signature of the function is obeyed. React spinned the same gun in the game of Russian roulette but was less lucky.
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#36Earlier quoted context omitted.
To be fair, this particular issue wouldn't have happened in C, Python, Forth, Zig, or a host of other languages. String-based weirdness is something of a JS issue.
This particular issue looks to be pretty uniquely a javascript problem. I don't even think hyper flexible languages like Ruby would ultimately experience this sort of problem.
Ruby can experience a similar problem[1], but that’s largely because its metaobject protocol draws no distinction between a read-only field and a zero-argument method. Python’s model does not have that issue (at the cost of significant complexity) and it is about as flexible as Ruby’s ultimately. (Python’s more rigid syntax is not relevant either way.)
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#37Earlier quoted context omitted.
This is like some kind of psy-op to get people to hate rust.
Sure. Then good companies will be able to filter out people by only hiring those who know Rust.
I don’t think this kind of sanctimoniousness has much of a success rate as a marketing approach.
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#38Earlier quoted context omitted.
Sure. Then good companies will be able to filter out people by only hiring those who know Rust.
I don’t think this kind of sanctimoniousness has much of a success rate as a marketing approach.
No marketing can save humanity from its inevitable demise.
Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#39Re: Property-Based Testing Caught a Security Bug I Never Would Have Found
#40Earlier quoted context omitted.
You forgot 0, -1, null, "1".
Since I work in a strongly typed languages the last two will fail to compile and are thus not worth the bother - those who don't have that luxury of course need to test the edge cases that apply to them. The first are maybe, in my experience they are rarely a problem, but we need to go from the abstract to the particular algorithm before we can have a discussion on if they are potentially a problem or not.
If you're using Rust you can push it one level further: use a NonZeroU*.
More than that and you'll need nightly for now: https://docs.rs/ranged_integers/latest/ranged_integers/