Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Six curl CVEs after OpenAI and Anthropic came back with zero

aisle.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#31

Post by zamadatix »

fweimer wrote: post #24
markasoftware wrote: post #17

Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)

As far as I understand it, the other efforts have not reported most of their findings to upstream developers, focusing on critical findings only. This is understandable because upstream interactions at scale are difficult.

In the case of big projects like curl the interaction seems a bit more complete. E.g. There are some other blog posts about how the engagements and reviews worked which go decently beyond a pre-filtered dump of high severity CVE claims appearing out of the blue.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#32

Post by bluGill »

goobreee wrote: post #20
bluGill wrote: post #12

OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed. If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then sev…

I think you might be misunderstanding this? This is, from my understanding, what went down: 1. curl was scanned by many different things, including AISLE, and many bugs were fixed so imho it was head-to-head, the very same codebase => it's a legit comparison

Curl has been scanned by mythos and several other AI tools several times over the past year already. Mythos found nothing this round, but when it was first released it found issues which have been fixed - and several other AIs had already scanned curl for issues which had been fixed by then.

We can say that this is a useful tool, but is it better or worse than the others - there is no way to make that conclusion.

Edit, wait, are you claiming that asile was already used and those issues fixed, and it still found more. That is valid, but it only says that asile is better by enough that is worth an upgrade, while the others probably are not. It is not valid in comparing to other tools. (assuming this is true, I've never heard of asile before this)

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#33

Post by lukeschlather »

matherial wrote: post #25
bch wrote: post #15

Earlier quoted context omitted.

Also sounds incredibly compute intensive.

Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.

This sounds like a swarm of agents with particular prompting that happens to guide the LLMs toward doing a lot of fuzzing, so it's not either/or; you're getting all the compute requirements of both.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#34

Post by bluGill »

zamadatix wrote: post #19
bluGill wrote: post #12

OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed. If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then sev…

"How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.

But that isn't what happened! Mythos has found issues in the past - which are now fixed (or so we should assume, I didn't verify but curl is very good about fixing issues). At most we can say the current version of mythos isn't better than the last version (a new version of mythos was just released, I'm not sure if that was even the one used in this scan)

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#36

Post by bluGill »

graemep wrote: post #23

Curl seems to becoming one of the favourite things to demo AI finding vulns. Curl is going to end up incredibly secure.

Curl has a well earned reputation for high quality code. If you find something there it means you are good. There is a lot of software where finding a vulnerability mostly means you bothered to look and are not completely stupid. Nobody is going to be impressed if you find an issue with something that everybody already knows is poorly coded.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#37

Post by tosti »

One does not "discover" a CVE like this. To discover a CVE would mean you searched for a particular piece of software and found it vulnerable according to the NVD. That's not a novel discovery by any means.

What they did is they found bugs and that they were exploitable in certain edge cases. As the bugs turned out to be vulnerabilities, they were assigned a CVE in the NVD with low severity.

IMHO Aisle stockedpiled too much in the marketing shelves.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#38

Post by mynameisash »

bluGill wrote: post #36
graemep wrote: post #23

Curl seems to becoming one of the favourite things to demo AI finding vulns. Curl is going to end up incredibly secure.

Curl has a well earned reputation for high quality code. If you find something there it means you are good. There is a lot of software where finding a vulnerability mostly means you bothered to look and are not completely stupid. Nobody is going to be impressed if you find an issue with something that everybody already knows is poorly coded.

> Curl has a well earned reputation for high quality code.

SQLite also has a very good reputation. I vaguely recall hearing about one SQLite vulnerability discovered via AI, but I thought it turned out to be a nothingburger. A quick search turned up CVE-2025-6965[0,1], published on 2025-07-15, which affects SQLite I'm not much of a security nerd, but my naive reading of this implies that it was already known and fixed as of the time of the CVE; in other words, the AI discovery didn't seem particularly helpful (though one could argue that it did successfully discover a CVE).

Has AI found many/any other vulnerabilities in SQLite?

[0] https://cybersecuritynews.com/sqlite-0-day-vulnerability/

[1] https://nvd.nist.gov/vuln/detail/cve-2025-6965

[2] https://sqlite.org/releaselog/3_50_2.html

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#39

Post by fc417fc802 »

bluGill wrote: post #32
goobreee wrote: post #20

Earlier quoted context omitted.

I think you might be misunderstanding this? This is, from my understanding, what went down: 1. curl was scanned by many different things, including AISLE, and many bugs were fixed so imho it was head-to-head, the very same codebase => it's a legit comparison

Curl has been scanned by mythos and several other AI tools several times over the past year already. Mythos found nothing this round, but when it was first released it found issues which have been fixed - and several other AIs had already scanned curl for issues which had been fixed by then. We can say that this is a useful tool, but is it better or worse than the others - there is no way to make that conclusion. Edi…

I don't understand your objection. If tool A says "job is done" and tool B says "found N additional tasks that need doing" how does the history matter? B is turning up things that A isn't thus B is performing better. They both had access to the same inputs here.

I suppose it's interesting to wonder if B would have turned up issue X which A previously found. But that seems largely academic to me. There is a code base right now with N known issues (thanks to B) and A is saying everything is good. It seems like that's all that should matter here.

Re: Six curl CVEs after OpenAI and Anthropic came back with zero

#40

Post by bluGill »

mynameisash wrote: post #38
bluGill wrote: post #36

Earlier quoted context omitted.

Curl has a well earned reputation for high quality code. If you find something there it means you are good. There is a lot of software where finding a vulnerability mostly means you bothered to look and are not completely stupid. Nobody is going to be impressed if you find an issue with something that everybody already knows is poorly coded.

> Curl has a well earned reputation for high quality code. SQLite also has a very good reputation. I vaguely recall hearing about one SQLite vulnerability discovered via AI, but I thought it turned out to be a nothingburger. A quick search turned up CVE-2025-6965[0,1], published on 2025-07-15, which affects SQLite I'm not much of a security nerd, but my naive reading of this implies that it was already known and fixe…

Hard to say what AI has been used in SQLite unless someone is talking. My impression is the maintainer doesn't talk about this type of thing much, but maybe I'm wrong.
Post reply on HN