Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Property-Based Testing Caught a Security Bug I Never Would Have Found

kiro.dev

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#11

Post by mirthturtle »

fireflash38 wrote: post #6
mananaysiempre wrote: post #2

TL;DR: obj[key] with user-controlled key == "__proto__" is a gift that keeps on giving; buy our AI tool that will write subtle vulnerabilities like that which you yourself won’t catch in review but then it will also write some property-based tests that maybe will

Don't forget you can use AI to turn a 50 word blog post into a 2,000 word one!

For real. The bullet-point summary at the beginning with a "Why this matters for..." immediately followed by, "This isn't just a theoretical exercise—it's a real example of..." Dead giveaways.

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#12

Post by oncallthrow »

sublinear wrote: post #4

> Is this exploitable? No. ... JSON.stringify knows to skip the __proto__ field. ... However, refactors to the code could ... [cause] subtle incorrectness and sharp edge cases in your code base. So what? This line of what-if reasoning is so annoying especially when it's analysis for a language like javascript. There's no vulnerability found here and most web developers are well aware of the risky parts of the languag…

> most web developers are well aware of the risky parts of the language

In my experience this really isn’t true. Most web developers I know are not familiar (enough) with prototype pollution.

By the way, this isn’t because they are “dumb”. It’s the tool’s fault, not the craftsman’s, in this case. Prototype pollution is complicated and surprising

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#13

Post by sevensor »

philipwhiuk wrote: post #5

> We write unit tests for the happy path, maybe a few edge cases we can imagine, but what about the inputs we'd never consider? Many times we assume that LLMs are handling these scenarios by default, Do we?

What’s interesting to me about this, reckless as it is, is that the conversation has begun to shift toward balancing LLMs with rigorous methods. These people seem to be selling some kind of AI hype product backed by shoddy engineering, and even they are picking up on the vibe. I think this is a really promising sign for the future.

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#14

Post by jgalt212 »

sublinear wrote: post #4

> Is this exploitable? No. ... JSON.stringify knows to skip the __proto__ field. ... However, refactors to the code could ... [cause] subtle incorrectness and sharp edge cases in your code base. So what? This line of what-if reasoning is so annoying especially when it's analysis for a language like javascript. There's no vulnerability found here and most web developers are well aware of the risky parts of the languag…

> insane false positives SAST scans dump on you

Great LLM use case: Please explain to the box ticking person why these "insane false positives SAST" are false and / or of no consequence.

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#15

Post by raphting »

fireflash38 wrote: post #6
mananaysiempre wrote: post #2

TL;DR: obj[key] with user-controlled key == "__proto__" is a gift that keeps on giving; buy our AI tool that will write subtle vulnerabilities like that which you yourself won’t catch in review but then it will also write some property-based tests that maybe will

Don't forget you can use AI to turn a 50 word blog post into a 2,000 word one!

Exactly this! AI fluff all over in that article.

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#17

Post by bpt3 »

philipwhiuk wrote: post #5

> We write unit tests for the happy path, maybe a few edge cases we can imagine, but what about the inputs we'd never consider? Many times we assume that LLMs are handling these scenarios by default, Do we?

When "we" = "developers we imagined when using LLMs to generate this marketing slop based on a contrived scenario", then sure!

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#19

Post by kittoes »

koakuma-chan wrote: post #18

This kind of bug would never have happened if you used a proper programming language like Rust.

This just can't be your answer to everything... the article clearly stated that they're developing a client application for browsers. Rust advocates like yourself are really doing more harm than good by ignoring real world constraints.

Re: Property-Based Testing Caught a Security Bug I Never Would Have Found

#20

Post by koakuma-chan »

kittoes wrote: post #19
koakuma-chan wrote: post #18

This kind of bug would never have happened if you used a proper programming language like Rust.

This just can't be your answer to everything... the article clearly stated that they're developing a client application for browsers. Rust advocates like yourself are really doing more harm than good by ignoring real world constraints.

You can use Rust to develop client application for browsers. Check out dioxus.
Post reply on HN