Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise of 'Vibe Hacking' Is the Next AI Nightmare

wired.com

Post reply on HN
Print view View original

41–50 of 56 posts

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#41

Post by cibyr »

bradyriddle wrote: post #9
tptacek wrote: post #4

Software security has been running off large-scale automation for over a decade. LLMs might or might not be a step change in that automation (I'm optimistic but uncertain), but, unlike in conventional software development, the standard arguments about craft and thoughtfulness aren't operative here. If there was an argument to be had, it would have been had around the time Google stood up the mega fuzzing farms. A fun…

Say more about these mega fuzzing farms. I haven't heard anything about this.

ClusterFuzz: https://google.github.io/clusterfuzz/

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#42

Post by blibble »

diggan wrote: post #40
blibble wrote: post #33

Earlier quoted context omitted.

your hardware won't last 100 years > And even over the course of 100 year, people will continue the machine learning science the weak point is big tech: without their massive spending the entire ecosystem will collapse so that's what we target, politically, legally, technologically and regulatory we (humanity) only need to succeed in one of these domains once, then their business model becomes nonviable once you cut…

I think you over-estimate how difficult it is to get "most of the world" to agree to anything , and under-estimate how far people are willing to go to make anything survive even when lots of people want that thing to die.

> I think you over-estimate how difficult it is to get "most of the world" to agree to anything

agreement isn't needed

its success sows the seeds of its own destruction, if it starts eating the middle class: politicians in each and every country that want to remain electable will move towards this position independently of each other

> and under-estimate how far people are willing to go to make anything survive even when lots of people want that thing to die.

the structural funding is such that all you need to do is chop off the funding from big tech

the nerd in their basement with their 2023 macbook is irrelevant

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#43

Post by jdefr89 »

tptacek wrote: post #35
jdefr89 wrote: post #28

Alright folks.. To qualify myself. I am a vulnerability Researcher @ MIT. My day to day research concerns embedded hardware/software security. Some of my current/past endeavors involve AI/ML integration and understanding just how useful it actually is for finding/exploiting vulnerabilities. Just last week my lab hosted a conference that included MIT folks and the outsiders we invite. One talk was on the current state…

I'm a vuln researcher too, and we just had an article here about another vuln researcher using o3 to find a zero-day remote Linux kernel vulnerability. And not in an especially human-directed way: they literally set up 100 runs of o3, using the 'simonw `llm` tool, and sifted through the results. I'm having trouble reconciling what you wrote here with that result. Also with my own experiences, not necessarily of findi…

I might be. Deepsleep also sort of found a bug, but you need to ask yourself… is it doing it better than tools we already have? Could a fuzzer have found that bug in less time? How far along did it really need to be pushed and also.. I have no doubts it probably trained on certain types of bugs for certain specific code bases.. Did they test its ability to find the same bug after applying a couple transforms that trip up the LLM? Can you link me to this article about o3? I have my doubts. I’d love to see the working exploit…

Also if you throw these models at enough code bases, they will probably get lucky a couple times.. So far every claim I have seen didn’t stand up to rigorous scrutiny. People find one bug then inflate their findings and write articles that would make you think they are far more affective than reality and I am tired of this hype.

CURL had to stop accepting bounties after it found nearly all of em were just AI generated nonsense…

Also I stated that they indeed provide very large gains in certain areas. Like writing a fuzz harness and reversing binaries. I am not saying they have absolutely no utility I am simply tired of grifters attempting to inflate their findings for clout. Shit has gotten out of control.

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#44

Post by tptacek »

jdefr89 wrote: post #43
tptacek wrote: post #35

Earlier quoted context omitted.

I'm a vuln researcher too, and we just had an article here about another vuln researcher using o3 to find a zero-day remote Linux kernel vulnerability. And not in an especially human-directed way: they literally set up 100 runs of o3, using the 'simonw `llm` tool, and sifted through the results. I'm having trouble reconciling what you wrote here with that result. Also with my own experiences, not necessarily of findi…

I might be. Deepsleep also sort of found a bug, but you need to ask yourself… is it doing it better than tools we already have? Could a fuzzer have found that bug in less time? How far along did it really need to be pushed and also.. I have no doubts it probably trained on certain types of bugs for certain specific code bases.. Did they test its ability to find the same bug after applying a couple transforms that tri…

But that's exactly what people were saying about fuzzer farms in the mid-2000s, in the belief that artisanal audits would always be the dominant means of uncovering bugs. The truth was somewhere in between (it's still humans, but working at a higher layer of abstraction than they were before) but the fuzzer people were hugely right.

If you can reliably get x% lucky finding vulnerabilities for Y$ cost, then you simply scale that up to find more vulnerabilities.

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#45

Post by jdefr89 »

tptacek wrote: post #44
jdefr89 wrote: post #43

Earlier quoted context omitted.

I might be. Deepsleep also sort of found a bug, but you need to ask yourself… is it doing it better than tools we already have? Could a fuzzer have found that bug in less time? How far along did it really need to be pushed and also.. I have no doubts it probably trained on certain types of bugs for certain specific code bases.. Did they test its ability to find the same bug after applying a couple transforms that tri…

But that's exactly what people were saying about fuzzer farms in the mid-2000s, in the belief that artisanal audits would always be the dominant means of uncovering bugs. The truth was somewhere in between (it's still humans, but working at a higher layer of abstraction than they were before) but the fuzzer people were hugely right. If you can reliably get x% lucky finding vulnerabilities for Y$ cost, then you simply…

I don’t recall anyone saying anything of the sort back then about fuzzing? Back then you could run the most basic fuzzer and find tons of bugs! Where did you see people complaining about fuzzers??

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#46

Post by byt3bl33d3r »

tptacek wrote: post #36
byt3bl33d3r wrote: post #25

I've written tailored offensive security tools and malware for Red Teams for around a decade and now work in the AI space. The argument that LLMs will enable "super powered" malware and that existing security solutions won't be able to keep up, is completely overblown. I see 0 evidence of this being possible with the current incarnation of "AI" or LLMs. "Vide coded" malware will be easier to detect if the people crea…

For the record I buy your argument about "vibe-coded malware"; this cycle of hype has been running since 1995 and Nowhere Man's "Virus Creation Lab". I am however fixated on the impact LLMs will have on vulnerability research, and what that will do to the ecosystem.

100% agree on the impact of it on research. It's pretty obvious that it'll accelerate 0day discovery but standard defense in depth strategies prepare you for 0day vulns against your org.

It will be extremely interesting to see how vulnerability discovery evolves with LLMs but the whole "sky is falling hide your kids" hype cycle is ludicrous.

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#47

Post by vouaobrasil »

rglover wrote: post #30
hluska wrote: post #12

Earlier quoted context omitted.

Rather, it’s many different types of software running on many different systems around the world, each funded by a different party with its own motives. This is no movie…

And every single one has a power switch. I get the general "too many variables" argument, but the idea that humans have no means of stopping any of these apps/systems/algorithms/etc if they get "out of control" (a farce in itself as it's a chat bot ) is ridiculous. It's very interesting to see how badly people want to be living in and being an active participant in a sci-fi flick. I think that's far more concerning t…

Hmm, good point. Also, when COVID struck, although it took some time, everyone collectively participated in staying home (more or less, I know some people didn't but the participating was vast). We can do the same if we choose.

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#48

Post by tptacek »

jdefr89 wrote: post #45
tptacek wrote: post #44

Earlier quoted context omitted.

But that's exactly what people were saying about fuzzer farms in the mid-2000s, in the belief that artisanal audits would always be the dominant means of uncovering bugs. The truth was somewhere in between (it's still humans, but working at a higher layer of abstraction than they were before) but the fuzzer people were hugely right. If you can reliably get x% lucky finding vulnerabilities for Y$ cost, then you simply…

I don’t recall anyone saying anything of the sort back then about fuzzing? Back then you could run the most basic fuzzer and find tons of bugs! Where did you see people complaining about fuzzers??

If you go digging through the blogosphere of the time you'll turn it up. I feel like this is ~2006?

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#49

Post by jdefr89 »

tptacek wrote: post #48
jdefr89 wrote: post #45

Earlier quoted context omitted.

I don’t recall anyone saying anything of the sort back then about fuzzing? Back then you could run the most basic fuzzer and find tons of bugs! Where did you see people complaining about fuzzers??

If you go digging through the blogosphere of the time you'll turn it up. I feel like this is ~2006?

Bro in 2006 there wasn't any blogosphere. You had IRC.. I can’t find anything of the sort and do you have a link to this discovery that you say was made via LLM?

Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare

#50

Post by tptacek »

jdefr89 wrote: post #49
tptacek wrote: post #48

Earlier quoted context omitted.

If you go digging through the blogosphere of the time you'll turn it up. I feel like this is ~2006?

Bro in 2006 there wasn't any blogosphere. You had IRC.. I can’t find anything of the sort and do you have a link to this discovery that you say was made via LLM?

Bro, I've been a practitioner since 1996 and ran a fucking security blog in 2006.
Post reply on HN