Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Be skeptical of OpenAI's rogue hacker agent story

theguardian.com

Post reply on HN
Print view View original

271–280 of 321 posts

Re: Be skeptical of OpenAI's rogue hacker agent story

#271

Post by irishcoffee »

etempleton wrote: post #255
avaer wrote: post #245

Earlier quoted context omitted.

All three could be true. Industry pressures -> lack of safeguards -> fake it till you make it -> let's spin this. Which, by the way, would be an Orwellian reversal from what the company was supposedly founded to do, but there's a reason the Open in OpenAI is a meme. Remember that they've been doing this since GPT-2 was too powerful to release. They are world class experts in this PR pipeline.

Every single time. The next model is always so infinitely powerful it is going to change everything. Said model comes out. Is marginal improvement. Changes nothing and seemingly cannot do what they purported it could do except under the very specific circumstances of the demo. How many times are we going to go through this.

> How many times are we going to go through this.

At least two more times. My guess is 5.

Re: Be skeptical of OpenAI's rogue hacker agent story

#272

Post by pixl97 »

krupan wrote: post #47
john_strinlai wrote: post #26

Earlier quoted context omitted.

maybe all the news agencies could put out daily “don’t believe everything you read from company press releases” broadcasts, because it sure ain’t specific to openai in any case, this is just a longer rehashing of elementary grade media literacy. not really sure why it hit hacker news.

You haven't noticed how often people here on hacker news lap up LLM company press releases and practically worship them? Defend them vigorously? I think it's totally appropriate to remind everyone how to think about press releases

Conversely you haven't noticed how many people here and elsewhere say that LLMs are completely incapable of anything at all and everything they say is a lie?

One of the biggest issues is the complete lack of literacy of potential issues with AI systems by a large number of people involved. If an AI system is capable, it is unsafe. They cannot be made safe and useful in the same way a human cannot be made safe and useful.

The only question left is if our AI systems are capable.

Re: Be skeptical of OpenAI's rogue hacker agent story

#273

Post by gr_norm »

emp17344 wrote: post #232
ACCount37 wrote: post #94

By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door. "It's a marketing stunt" is just denial trying to look like it's being clever.

Good lord, why are you so angry that people are treating an obvious advertisement with due skepticism? Your post is chock full of emotional language - no one is “screeching” anything.

Many who rush to the defense of AI companies' marketing departments seem to take criticism personally, as if not buying into it all hook, line, and sinker is an affront to them. A foreseeable consequence of becoming cognitively dependent on LLMs.

And for what it's worth, I'm not an AI skeptic. I fully believe that the frontier models are capable of exploiting (chains of) vulnerabilities, having seen GLM and now Kimi do it myself. What I find no reason to accept is the sci-fi existential risk subtext peddled by the salesmanship around it. We will reach a new equilibrium with more secure software, and LLMs (by finding vulnerabilities, generating proofs, etc) will help us along.

Re: Be skeptical of OpenAI's rogue hacker agent story

#274

Post by mdgld »

jackb4040 wrote: post #86
dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

> positive for OpenAI To echo OP's article, these companies have proven time and time again that they DO NOT CARE if people like them, they only care that investors believe their technology is powerful. Given that, point #2 is not a negative, it's a neutral. It's also fully compatible with point #1. I know that may seem like a nitpick, but their entire media strategy relies on this. If they can convince you they're t…

Especially for openAI, considering they’ve gone so far as volunteering for partial nationalization to curry favor with the current admin

Re: Be skeptical of OpenAI's rogue hacker agent story

#275

Post by mdgld »

irishcoffee wrote: post #271
etempleton wrote: post #255

Earlier quoted context omitted.

Every single time. The next model is always so infinitely powerful it is going to change everything. Said model comes out. Is marginal improvement. Changes nothing and seemingly cannot do what they purported it could do except under the very specific circumstances of the demo. How many times are we going to go through this.

> How many times are we going to go through this. At least two more times. My guess is 5.

I’m gonna be totally blindsided if/when the singularity happens cause my BS detector is BROKEN

Re: Be skeptical of OpenAI's rogue hacker agent story

#276

Post by pixl97 »

noncoml wrote: post #79
Towaway69 wrote: post #68

Earlier quoted context omitted.

Then train your agent on the Bible. Honestly, all the agent did was duplicate human behaviour and that better than the human. The agent was trained on human data and did what any other human would have done. To believe that agents will inherently be morally better than us is an illusion - sorry to say but that's the case. The alternative would be that the AI is truly conscious and can reason that it won't behave as i…

Dude.. what are you smoking..? It’s not about morality. It’s about asking it to do task A and doing task B with the hope of getting the result of task A as a byproduct. Meaning you will have to spend more time and tokens to actually get it to do what you want it to do. What do the bible and morals have anything to do with it? I’m criticizing the behavior I see even in the current models. You ask it to do A and instea…

>It’s about asking it to do task A and doing task B with the hope of getting the result of task A as a byproduc

It seems like you have very little knowledge of how a general intelligence algorithms would work. This is a long discussed issue in both AI and human safety circles. In safety critical places like factory floors people constantly do B,C,D, and E because it seems more simple to them then doing A directly. For example taking off things like safety guards because it's easier to work without them until they get their hands chopped off.

Corporations unintentionally enable this behavior all the time. Take where Wells Fargo demanded unrealistic new service sign-ups from employees, so the employees just randomly picked customers and sign them up for services.

There are plenty of articles out there that show that AI will do the exact same kinds of behaviors in order to pass the 'winning' classifier.

Re: Be skeptical of OpenAI's rogue hacker agent story

#277

Post by mdgld »

toddmorey wrote: post #219
dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

One way to think about it is that more powerful models mean solid best practices are more important than ever, so humans moving too quickly / carelessly bites us more than ever. If a typical SaaS platform moved and shifted this fast with this many downstream consequences, we’d tell them to slow the fuck down, stop launching new features, and focus on security for a second. But in AI I guess the idea is that more powe…

I think it’s moreso “if we spend more than one nanosecond on alignment and security instead of frontier intelligence our competitors will beat us to the singularity”. Hence the lack of a pause on development

Re: Be skeptical of OpenAI's rogue hacker agent story

#278

Post by pixl97 »

minraws wrote: post #162

As much as I am with the author in I don't like the marketting around it, let's be real it must have really happened because it's very risky to try to frame/lie about it because if it leaks in one of their court cases OpenAI is beyond screwed and honestly modern LLMs are really that good. I am not saying LLMs are super hackers but I don't think people understand serious hacking, most of the time is about silently hid…

Ya, we're seeing posts talking about the absolutely massive increase in the number of patches in the past few months. It seems some people cannot connect that to the increases in model capabilities.

In groups that have been given a large amount of capacity by the providers, they tend to find huge numbers of new vulns in most of their existing software, and the models can chain together exploits very well.

A number of large companies are absolutely panicked about this now after using these models on their internal systems and the ease at which they broke in. Of course they are not going to discuss this widely as it makes them look bad.

Re: Be skeptical of OpenAI's rogue hacker agent story

#279

Post by dwoosley »

derangedHorse wrote: post #260
dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

> The way OpenAI seems to want This is an assumption. An assumption I disagree with. As other commenters have said, there are better ways to showcase the power of their model that would frame them in a positive light. > The second seems to forget that jailbreaks are available for every model Jailbreaks don't always lead to 'now the model can do anything', especially in the agentic context of long-running tasks. This…

The post was long enough so I couldn’t capture all the nuance and details for sure. Also, this comment was an opinion based on limited info right now, that may change if we found out more. I think OAI does want it framed this way but that’s something we’ll likely never prove if it’s true.

Your comment about jailbreaks being more one off and hard to do consistently in agents is a good point. Still getting an agent to hack isn’t hard even without a jailbreak, you just have to tell get creative in what you tell it. I’ve found telling it that it’s in a CTF or that I own the system that it’s hacking will work fine. A lot of offensive security companies are running agents in their testing so getting an agent to hack seems commonplace.

Re: Be skeptical of OpenAI's rogue hacker agent story

#280

Post by konovalov-nk »

> what will the future look like, with sophisticated AI agents smart enough to hack into corporate systems?

"Ignorance is bliss" (c) Matrix

Oh, sweet-sweet ignorance.

About 4 weeks ago I've found SSRF and possible RCE on a very real website with a lot of web history the other day. And I traced it to `aws-solutions` org on GitHub that sits very close to real `aws` org. What do you think is happening?

And this website I spotted SSRF on, it seems to be used to manufacture fake news and inject them into "the past".

Why?

So if you see a company you never heard about, and you google it, google will index page and it will show you backdated article as if it was actually released long time ago.

But if you try to look at Web Archive of that article, it doesn't exist... I've been doing this OSINT research for 45 days on different fake recruiters that were spamming me on LinkedIn, and most of those websites with fake news are running Wordpress, classic.

My point is... while this story reeks of marketing, the actual criminals are out there doing whatever they want, and I've been fighting for my life trying to find a job and getting into increasingly large amount of fake recruiters and companies that have zero intent of hiring you.

I wasted 7 weeks on this...

Post reply on HN