Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Six curl CVEs after OpenAI and Anthropic came back with zero

aisle.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#21

Post by goobreee »

thih9 wrote: post #16
bluGill wrote: post #12

OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed. If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then sev…

I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.

i don't think this is doable fairly. as they say in the blog post, the only fair way to is to look for new, previously undiscovered zero-days, otherwise you always risk the model has in some way been trained on the vulnerabilities. looking for legit new stuff is the only way to prevent leakage (even accidental one)

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#22

Post by goobreee »

markasoftware wrote: post #17

Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)

i don't think this is correct. if you look at this article by the curl founder daniel stenberg (https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v...), he talks about how he previously ran Mythos on curl and that it found 5 issues: 1 turned out to be a low severity CVE, 3 were false positives, and 1 just a bug. So a) Mythos detects low severity CVEs too, and b) it is fairly noisy

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#24

Post by fweimer »

markasoftware wrote: post #17

Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)

As far as I understand it, the other efforts have not reported most of their findings to upstream developers, focusing on critical findings only.

This is understandable because upstream interactions at scale are difficult.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#25

Post by matherial »

bch wrote: post #15
tux3 wrote: post #8

Earlier quoted context omitted.

The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.

Also sounds incredibly compute intensive.

Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#27

Post by vorticalbox »

melvinroest wrote: post #3

Wow, this announcement is good content marketing. Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't. It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities. >…

It defaults to gpt5.4 nano

https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#28

Post by vorticalbox »

drdrd wrote: post #4
melvinroest wrote: post #3

Wow, this announcement is good content marketing. Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't. It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities. >…

> what models the use under the hood Presumably their own, wouldn’t they?

Default to gpt 5.4 nano

https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#29

Post by bryanlarsen »

vorticalbox wrote: post #27
melvinroest wrote: post #3

Wow, this announcement is good content marketing. Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't. It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities. >…

It defaults to gpt5.4 nano https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...

A repo named "nano-analyzer" unsurprisingly uses gpt5.4 nano. I doubt their "pay them money" version uses nano.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#30

Post by zamadatix »

bch wrote: post #15
tux3 wrote: post #8

Earlier quoted context omitted.

The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.

Also sounds incredibly compute intensive.

Fuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.
Post reply on HN