151–160 of 181 posts
Earlier quoted context omitted.
You literally cannot, since ANY changes to code tend to introduce unintended (or at least not explicitly requested) new behaviors.
Eventual convergence? Assuming each defect fix has a 30% chance of introducing a new defect, we keep cycling until done?
Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link. The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-fi…
But it is true that if attackers more easily find holes in software that's hard to patch (embedded), that's a problem.
Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link. The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-fi…
The higher the productivity multiplier towards exploiting software, the more developers would find themselves severely outmatched: exploiting software is someone’s full-time job, whereas the engineers already have one—building it. To express this in numerical terms, let’s consider developer’s incentive to spend effort learning to find and actually finding vulnerabilities in their software (as oppposed to building it)…
But the attackers needs to spread their attack over many products, while the engineers only need to defend one.
> The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10.
Except that's true even without LLMs. LLMs improve both sides' capabilities by the same factor (at least hypothetically).
> Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose.
I'm not sure that's right, because once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them. So again, we can remove the "LLM factor" from both sides of the equation. If anything, security skills can spread more easily to defenders with LLM because without LLMs, the security skill of the attackers require more effort to develop.
Earlier quoted context omitted.
The higher the productivity multiplier towards exploiting software, the more developers would find themselves severely outmatched: exploiting software is someone’s full-time job, whereas the engineers already have one—building it. To express this in numerical terms, let’s consider developer’s incentive to spend effort learning to find and actually finding vulnerabilities in their software (as oppposed to building it)…
> exploiting software is someone’s full-time job, whereas the engineers already have one—building it. But the attackers needs to spread their attack over many products, while the engineers only need to defend one. > The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can d…
> But the attackers needs to spread their attack over many products, while the engineers only need to defend one.
Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely.
Realistically, you have people whose job or passion is to develop software, who often work not on one but on N projects at the same time (especially in OSS), and who definitely aren’t going to make finding vulnerabilities their full-time job because if they do then there’ll be no one to build the thing in the first place.
> Except that's true even without LLMs.
Of course. That’s why I put it before I started taking into account LLMs. LLMs multiply the pre-existing imbalance.
> once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them
Sure, that’s an interesting point. I’m sure the attackers try to conceal their methods; the way we tend to find out about it is when an exploit is exhausted, stops being worth $xxxxxxxx, and starts to be sold on mass markets, at which point arguably it’s a bit late. Furthermore, you still mention those mystical “defenders”, as if you would expect an average software project to have any dedicated defenders.
(Edited my reply to the latest point, I didn’t read it correctly the first time.)
Earlier quoted context omitted.
That might be one outcome, especially for large, expertly-staffed vendors who are already on top of this stuff. My real interest in what happens to the field for vulnerability researchers.
Perhaps a meta evolution, they become experts at writing harnesses and prompts for discovering and patching vulnerabilities in existing code and software. My main interest is, now that we have LLMs, will the software industry move to adopting techniques like formal verification and other perhaps more lax approaches that massively increase the quality of software.
Earlier quoted context omitted.
If that was a jab it my writing then yes, I am absolutely being sincere because I am an expert on this topic. LLMs went from being ok at one-shoting a function a to being so good at hacking that it's difficult to evaluate them. Prospective customers get back to us after a demo and tell us about the exploits it found on their services that are so vague and technical that they wouldn't think to look for them.
> Prospective customers get back to us after a demo and tell us about the exploits it found on their services that are so vague and technical that they wouldn't think to look for them. Um, have you actually verified that those are actual exploits then? Vague and technical sounds exactly like a description of AI slop...
Earlier quoted context omitted.
It's this talk right here: https://www.youtube.com/watch?v=1sd26pWhfmg 7 minutes in, he shows the SQLI he found in Ghost (the first sev:hi in the history of the project). If I'd remembered better, I would have mentioned in the post: * it's a blind SQL injection * Claude Code wrote an exploit for it. Not a POC. An exploit.
> Not a POC. An exploit. What's the distinction? A proof of concept is just something that demonstrates that a bug is possible to exploit, by doing so.
POC generally means “you can demonstrate unintentional behavior”.
“Exploit” means you can gain access or do something malicious.
It’s a fine line. Author’s point is that the LLM was able to demonstrate some malfeasance, not just unintended consequence. That’s a big deal considering that actual malicious intent generally requires more knowhow than raw POC.
Earlier quoted context omitted.
Assuming you can catch every new bug it introduces. Both assumptions being unlikely. You also end up with a code base you let an AI agent trample until it is satisfied; ballooned in complexity and redudant brittle code.
You can have an AI agent refactor and improve code quality.
Now I used the agent to do a lot of the grunt work in that refactor, but it was still a design decision initiated by me. The chatbot, left unattended, would not have seen that needed to be done. (And when, during my refactor, it tried to fold in the fourth case I had to stop it.)
(And for a lot of code, that's ok - my static site generator is an unholy mess at this point, and I don't much care. But for paid work...)
I don't understand why the takeaway here is (unless I'm missing something), more or less "everything is going to get exploited all the time". If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software (or, at the very least, software for which LLMs can no longer find any new vulnerabilities)?
Earlier quoted context omitted.
You can have an AI agent refactor and improve code quality.
But, have you any code that has been vetted and verified to see if this approach works? This whole Agentic code quality claim is an assertion, but where is the literal proof?