We are no closer to this than "vibe-businessing", where you vibe your way into a profit-generating business powered by nothing than AI agents.
The Rise of 'Vibe Hacking' Is the Next AI Nightmare
31–40 of 56 posts
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#32Software security has been running off large-scale automation for over a decade. LLMs might or might not be a step change in that automation (I'm optimistic but uncertain), but, unlike in conventional software development, the standard arguments about craft and thoughtfulness aren't operative here. If there was an argument to be had, it would have been had around the time Google stood up the mega fuzzing farms. A fun…
rms@gnu.ai.mit.edu It was actually the A.I. Lab at M.I.T. and they already had their own dedicated subdomain for it. This had to have been around 1990-91. And IIRC, the actual admins made a valiant effort to keep all the shell users away from "root" privileges, so it wasn't a total dumpster fire and the system stayed alive, mostly https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artif...
ITS had no root
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#33Earlier quoted context omitted.
the nvidia/AMD/apple chips all require proprietary firmware blobs, it can be enforced in there yes you won't get people that won't ever update, but you'll get the overwhelming majority and the hardware the never-updaters use will eventually fail and won't be able to be replaced also: ban the release of new "open" models, they will slowly become out of date and useless combine these, and the problem will solve itself…
> they will slowly become out of date and useless Models released today are already useful for a bunch of stuff, maybe over the course of 100 year they could be considered "out of date", but they don't exactly bitrot by themselves because they sit on a disk, not sure why'd they suddenly "expire" or whatever you try to hint at. And even over the course of 100 year, people will continue the machine learning science, re…
your hardware won't last 100 years
> And even over the course of 100 year, people will continue the machine learning science
the weak point is big tech: without their massive spending the entire ecosystem will collapse
so that's what we target, politically, legally, technologically and regulatory
we (humanity) only need to succeed in one of these domains once, then their business model becomes nonviable
once you cut off the snake's head, the body will die
the boosters in search of a quick buck will then move onto the next thing (probably "quantum")
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#34Software security has been running off large-scale automation for over a decade. LLMs might or might not be a step change in that automation (I'm optimistic but uncertain), but, unlike in conventional software development, the standard arguments about craft and thoughtfulness aren't operative here. If there was an argument to be had, it would have been had around the time Google stood up the mega fuzzing farms. A fun…
rms@gnu.ai.mit.edu It was actually the A.I. Lab at M.I.T. and they already had their own dedicated subdomain for it. This had to have been around 1990-91. And IIRC, the actual admins made a valiant effort to keep all the shell users away from "root" privileges, so it wasn't a total dumpster fire and the system stayed alive, mostly https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artif...
I mean, I remember, in 1994, being on those systems. But it meant nothing. Anybody could be. There wasn't even a glimmer of interestingness about it. It was like "ls"'ing around an anonymous FTP server.
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#35Alright folks.. To qualify myself. I am a vulnerability Researcher @ MIT. My day to day research concerns embedded hardware/software security. Some of my current/past endeavors involve AI/ML integration and understanding just how useful it actually is for finding/exploiting vulnerabilities. Just last week my lab hosted a conference that included MIT folks and the outsiders we invite. One talk was on the current state…
I'm a vuln researcher too, and we just had an article here about another vuln researcher using o3 to find a zero-day remote Linux kernel vulnerability. And not in an especially human-directed way: they literally set up 100 runs of o3, using the 'simonw `llm` tool, and sifted through the results.
I'm having trouble reconciling what you wrote here with that result. Also with my own experiences, not necessarily of finding kernel vulnerabilities (I haven't had any need to do that for the last couple years), but of rapidly comprehending and analyzing kernel code (which I do need to do), and realizing how potent that ability would have been on projects 10 years ago.
I think you're wrong about this.
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#36I've written tailored offensive security tools and malware for Red Teams for around a decade and now work in the AI space. The argument that LLMs will enable "super powered" malware and that existing security solutions won't be able to keep up, is completely overblown. I see 0 evidence of this being possible with the current incarnation of "AI" or LLMs. "Vide coded" malware will be easier to detect if the people crea…
For the record I buy your argument about "vibe-coded malware"; this cycle of hype has been running since 1995 and Nowhere Man's "Virus Creation Lab". I am however fixated on the impact LLMs will have on vulnerability research, and what that will do to the ecosystem.
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#37We are no closer to this than "vibe-businessing", where you vibe your way into a profit-generating business powered by nothing than AI agents.
(Looks around)
You know, there are some pretty crazy run rates out there.
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#38Earlier quoted context omitted.
rms@gnu.ai.mit.edu It was actually the A.I. Lab at M.I.T. and they already had their own dedicated subdomain for it. This had to have been around 1990-91. And IIRC, the actual admins made a valiant effort to keep all the shell users away from "root" privileges, so it wasn't a total dumpster fire and the system stayed alive, mostly https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artif...
I mean, I remember, in 1994, being on those systems. But it meant nothing. Anybody could be. There wasn't even a glimmer of interestingness about it. It was like "ls"'ing around an anonymous FTP server.
Hey, I cannot even begin to describe the thrill I got when I first found my way to the AF.MIL anon-ftp server! It was probably sparsely populated with public domain software and a couple boring games, but it felt like I'd just walked in the front gate of Miramar and witnessed the Blue Angels doing barrel rolls.
Sure, it was basically "a poster on the wall" for the US Air Force, and the US Army guy on Usenet shared nothing about his actual Ballistics Research Labs experiments, but for a college freshman kid, I'd never been on a way k00ler bboard, doodz!!1
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#39The AI nightmare after that is "vibe capitalism". You put in a pitch deck and an operating business comes out. Somebody should pitch that to YC.
this already exists, it's called a venture capital fund
Re: The Rise of 'Vibe Hacking' Is the Next AI Nightmare
#40Earlier quoted context omitted.
> they will slowly become out of date and useless Models released today are already useful for a bunch of stuff, maybe over the course of 100 year they could be considered "out of date", but they don't exactly bitrot by themselves because they sit on a disk, not sure why'd they suddenly "expire" or whatever you try to hint at. And even over the course of 100 year, people will continue the machine learning science, re…
your hardware won't last 100 years > And even over the course of 100 year, people will continue the machine learning science the weak point is big tech: without their massive spending the entire ecosystem will collapse so that's what we target, politically, legally, technologically and regulatory we (humanity) only need to succeed in one of these domains once, then their business model becomes nonviable once you cut…
I think you over-estimate how difficult it is to get "most of the world" to agree to anything, and under-estimate how far people are willing to go to make anything survive even when lots of people want that thing to die.