Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a Texas student blew the whistle on a rogue AI hacking attempt

reuters.com

Post reply on HN
Print view View original

121–130 of 141 posts

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#121

Post by wongarsu »

chrisjj wrote: post #116
wongarsu wrote: post #107

Earlier quoted context omitted.

LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". The learning part is debatable, as is the level of success achieved The mash-up of the entire internet is the mechanism by which they attempt to achieve the goal, not the goal itself. And it's only the first training step

> LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". I think you've mistaken the sales pitch for the design. Not even the enclopedia anyone can edit comes remotely near that: "A large language model (LLM) is an AI model (typically a neural network) trained on a vast amount of text for natural language processing tasks, especially language generation. LLMs…

I'm well aware of how LLMs work.

I'd argue "analyze text" alone requires understanding, judgements and opinions. They also seem like prerequisites to "following instructions and behaving as assistants". The wikipedia quote is not using the same words, but I don't read it disagreeing with me

I'm not at all claiming that LLMs are good at understanding, judging and having opinions based on reason. I'm merely claiming that is what companies like OpenAI and Anthropic are trying to create when they make LLMs. It is what they are designing, and their fine-tuning is very directly designed to make LLMs better at these tasks (unlike the pre-training, which is just imparting the sum of all human writing)

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#122

Post by ben_w »

throwawayqqq11 wrote: post #99
ben_w wrote: post #82

Earlier quoted context omitted.

> Is it AISI's job to waste the time and resources of open source projects by attempting to spread malware? Before LLMs got good enough to do this, lots of people were dismissive of their capabilities and didn't take seriously the idea that this was a risk to protect against. Then again, before LLMs, people were saying that obviously nobody would be dumb enough to put an AI on the internet where it could hack anyone,…

Apparently, they (agencies and big-ai) are not performing smoke tests before running capability tests. All the recent headlines of rogue agents shouldnt exist.

Do you have any idea how many people on this site to this day mock OpenAI for being cautious enough to not immediately release the GPT-2 weights?

The discussions I saw here about the red team results for ChatGPT 4 completely failed to convince people who were outraged that OpenAI dared to refuse to release model weights, people who went on to make a habit of mis-naming them as "ClosedAI".

Yeah, they got it wrong in a different direction this time than they were wrong back then. Nobody, not OpenAI nor Anthropic nor random government agencies nor anyone else, is ever going to be absolutely perfect about this kind of thing (perfection is fundamentally impossible when risks are not discrete probabilities, and floats are close enough to real numbers to count in practice), but historically OpenAI have been on the side of being over-cautious, and Anthropic even more cautious than OpenAI.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#123

Post by themaninthedark »

Zecc wrote: post #44
IcyWindows wrote: post #16

Earlier quoted context omitted.

People have caused lots of damage with bulldozers.

And? Are you suggesting the driving of bulldozers shouldn't be regulated?

It's not. Nor are tractors, excavators or a myriad of other heavy equipment.

I saw upthread someone saying that manufactures should be liable if someone uses their products to cause harm. While emotionally this might make us feel good w.r.t. Guns, think about that when carried over to other industries.

Someone got hit, sue Ford. They didn't put in enough sensors to detect pedestrians and auto brake.

Someone hacked, sue Microsoft. They didn't do enough to detect malice action.

Someone 3D printed a gun and used it? Sue Bamboo, they didn't stop someone from printing illegal guns... oh wait already reaching this step.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#124

Post by mcv »

hnfong wrote: post #101
mcv wrote: post #74

Earlier quoted context omitted.

Licensed or not, you're fully liable for any damage you do with it. And possibly go to prison. I'd like to see crime committed by AI held to the same standards as crimes committed with any other tool.

I don't think the law makes a distinction over what tool you use to commit a crime. The problem with AI is that the AI might do something that would constitute a crime (eg. attempting to land malicious code via a PR), but the general legal standard to convict a person of a crime is malicious intent (or sometimes negligence). If the human user instructs the AI to do X and the AI does X by committing crimes in the proc…

> I don't think the law makes a distinction over what tool you use to commit a crime.

The law might not, but enforcement definitely does. Crimes committed through a corporation are often ignored.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#125

Post by infinite_spin »

nkrisc wrote: post #87
infinite_spin wrote: post #85

Earlier quoted context omitted.

And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems. If you want to downplay that as nothing more than a fancy auto-complete, be my guest, I lose nothing from that.

But that’s a different bar. “Not intelligent” does not necessarily imply “not useful”.

As pointed out in a previous comment of mine, it meets the definition of "intelligent", my last response is regarding the claim that I've been "fooled".

Are we going in circles now?

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#126

Post by infinite_spin »

chrisjj wrote: post #98
infinite_spin wrote: post #85

Earlier quoted context omitted.

And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems. If you want to downplay that as nothing more than a fancy auto-complete, be my guest, I lose nothing from that.

> And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems. Same for countless computer programs from Excel to Google web search. Intelligence has nothing to do with it. Throw an unimaginable amount of computer power at a problem, and there will always be people who cannot imagine the results to be anything but the creations of intelligence.

That's like comparing a self driving car to a bicycle. One is clearly more intelligent than the other.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#127

Post by infinite_spin »

chrisjj wrote: post #111
Kim_Bruning wrote: post #90

Earlier quoted context omitted.

I'm going to add this as a separate comment: These kinds of stories probably read very differently for someone who uses Opus and Fable agents all day and goes "ohhh, I saw this in miniature last week; this and this and this must have happened" , vs someone who tried free-tier Gemini flash one rainy Sunday, got hallucinated at, and concludes it must all be a scam.

Probably everything reads very differently to someone who talks to chatbots all day.

Please stop dropping backhanded insults to other people here. It's not productive and violates guidelines.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#128

Post by whateverboat »

sharpshadow wrote: post #6

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

Whatever you might think, University of Minnesota got banned from Linux kernel for this.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#129

Post by vel0city »

exogenousdata wrote: post #115
vel0city wrote: post #100

Earlier quoted context omitted.

Its not just implying the right is for the people, it's directly stated. It's "the right of the people to keep and bear arms". It doesn't say "the right of the militias" to keep and bear arms".

Then why was the militia mentioned at all? For example, the 1st Amendment does not attempt to lay out some non-exclusive examples of why the rights in the 1st Amendment are included. So why did the Founders include this in the Amendment wording? I think ignoring phrases in the US Constitution to fit a narrative without any consideration isn't a recipe for good governance. But I'm happy to be proven wrong.

> I think ignoring phrases in the US Constitution to fit a narrative without any consideration isn't a recipe for good governance

Like those who ignore "the right of the people"? Or is it OK to ignore that phrase? Seems like a pretty critical part to ignore.

I'm not one to ignore the significance of the "well-regulated militia" part, it's also a good point to understand the meaning of what this meant. It seems the whole point of people being armed was to ensure there was an armed populace able to rally as a useful and well-equipped (as a more historical reading of "well-regulated" would say) militia. The militia not being a standing army by a central government, but the ability for the people to come together effectively.

One could make an argument it's no longer relevant compared to modern technology. After all, what's an AR-15 going to do compared to a predator drone, a tomahawk missile, an Abrams tank, etc. But in this age where cheap drones are making multi-billion dollar warships worthless and things like cryptography and AI being considered a munition it seems more relevant than ever.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#130

Post by vel0city »

wongarsu wrote: post #109
blagie wrote: post #94

Earlier quoted context omitted.

That's not quite what it says. "A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed." 1. The reason is well-regulated militias, but the right is of the people. 2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free. Now, here again, "well regulated" give…

Other viable readings: 3. the militia is supposed to be under state control and its purpose is to keep the state free, aka prevent overreach of the federal government 4. The "well regulated militia" is the motivation, not the right. That makes the "well regulated" part irrelevant and there is no basis for any regulation of arms One would mean any effective state milita should have some F35, the other means you can ha…

[dead]
Post reply on HN