Marketing Slop
Six curl CVEs after OpenAI and Anthropic came back with zero
11–20 of 68 posts
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#12OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.
If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#13Wow, this announcement is good content marketing. Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't. It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities. >…
It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#14This is an ad. I didn't learn anything from reading it.
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#15Wow, this announcement is good content marketing. Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't. It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities. >…
The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.
Also sounds incredibly compute intensive.
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#16OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed. If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then sev…
I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#17Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#18Earlier quoted context omitted.
> what models the use under the hood Presumably their own, wouldn’t they?
You mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.
Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#19OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed. If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then sev…
"How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.
Re: Six curl CVEs after OpenAI and Anthropic came back with zero
#20OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed. If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then sev…
I think you might be misunderstanding this? This is, from my understanding, what went down:
1. curl was scanned by many different things, including AISLE, and many bugs were fixed so imho it was head-to-head, the very same codebase => it's a legit comparison