Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scamlexity: When agentic AI browsers get scammed

guard.io

Post reply on HN
Print view View original

11–20 of 198 posts

Re: Scamlexity: When agentic AI browsers get scammed

#11

Post by OtherShrezzing »

WJW wrote: post #8
benterix wrote: post #5

This should hit the headlines. I was always of the opinion that AI of all kinds is not a threat unless someone decides to connect it to an actuator so that it has direct and uncontrolled effect on the external world. And now it's happening en masse with agents, MCPs etc. I don't even mention things we don't know about (military and other classified projects).

You don't have to guess about the military applications, it's all over the news. Even bog standard FPV drones that Ukraine is churning out at a rate of >100k/month have image recognition these days, so that if the video stream gets jammed they can finish off the mission autonomously. Even on a hobby level, ardupilot+openCV+cheap drone kit from amazon is a DIY project within the skill set of a significant part of the…

> so that if the video stream gets jammed they can finish off the mission autonomously.

The streams mostly don't get jammed anymore, because the low-cost FPV drones are physically connected to the ground by a long optical cable. The extent of their autonomous dangers are limited by the amount of fibre-optic cable left in the spool when they take off.

Re: Scamlexity: When agentic AI browsers get scammed

#13

Post by zenoprax »

hliyan wrote: post #7
jtc331 wrote: post #4

I appreciate that the article correctly points out the core design flaw here of LLMs is the non-distinction between content and commands in prompts. It’s unclear to me if it’s possible to significantly rethink the models to split those, but it seems that that is a minimal requirement to address the issue holistically.

Ah, it's like the good old days when operating systems like DOS didn't really make the distinction between executable files and data files. It would happily let you run any old .exe from anywhere on Earth. Viruses used to spread like wildfire until Norton Antivirus came along.

How is `curl virus.sh | bash` or `irm virus.ps | iex` any different?

Re: Scamlexity: When agentic AI browsers get scammed

#15

Post by cjonas »

JCM9 wrote: post #9

“Agentic” seems the be some quick pivot buzzword that the AI grifters started pushing as soon as generic AI started to show cracks. “Hey this AI stuff looks a bit overhyped.” “AI? Oh that’s kids stuff, let me tell you about our agentic features!” Giving flaky shaky AI the ability to push buttons and do stuff. What could possibly go wrong? Malicious actors will have a field day with this.

If you only give the AI the ability to do what the end user can already do, the risk is extremely low. It's essential no different then building a static web app where the client is connected to API for all operations. It basically just becomes a new way to interface into a application.

However... That's not how a lot of people are building. Giving an agentic system sensitive information (like passwords, credit cards) and then opening it up to the entire internet as a source for input as asking for your info to be stolen. It'd be like asking your grandma with dementia to manage all your email and online banking.

Re: Scamlexity: When agentic AI browsers get scammed

#16

Post by yorwba »

jtc331 wrote: post #4

I appreciate that the article correctly points out the core design flaw here of LLMs is the non-distinction between content and commands in prompts. It’s unclear to me if it’s possible to significantly rethink the models to split those, but it seems that that is a minimal requirement to address the issue holistically.

The flaw isn't just in the design, it's in the requirements. People want an AI that reads text they didn't read and does the things the text says need to be done, because they don't want to do those things themselves. And they don't want to have to manually approve every little action the AI takes, because that would be too slow. So we get the equivalent of clicking "OK" on every dialog that pops up without reading it, which is also something that people often do to save a bit of time.

Re: Scamlexity: When agentic AI browsers get scammed

#17

Post by victorbjorklund »

OtherShrezzing wrote: post #11
WJW wrote: post #8

Earlier quoted context omitted.

You don't have to guess about the military applications, it's all over the news. Even bog standard FPV drones that Ukraine is churning out at a rate of >100k/month have image recognition these days, so that if the video stream gets jammed they can finish off the mission autonomously. Even on a hobby level, ardupilot+openCV+cheap drone kit from amazon is a DIY project within the skill set of a significant part of the…

> so that if the video stream gets jammed they can finish off the mission autonomously. The streams mostly don't get jammed anymore, because the low-cost FPV drones are physically connected to the ground by a long optical cable. The extent of their autonomous dangers are limited by the amount of fibre-optic cable left in the spool when they take off.

Optical fiber drones is still the minority of drones (both because more expensive and because it has other downsides than radio)

Re: Scamlexity: When agentic AI browsers get scammed

#18

Post by cjonas »

cjonas wrote: post #15
JCM9 wrote: post #9

“Agentic” seems the be some quick pivot buzzword that the AI grifters started pushing as soon as generic AI started to show cracks. “Hey this AI stuff looks a bit overhyped.” “AI? Oh that’s kids stuff, let me tell you about our agentic features!” Giving flaky shaky AI the ability to push buttons and do stuff. What could possibly go wrong? Malicious actors will have a field day with this.

If you only give the AI the ability to do what the end user can already do, the risk is extremely low. It's essential no different then building a static web app where the client is connected to API for all operations. It basically just becomes a new way to interface into a application. However... That's not how a lot of people are building. Giving an agentic system sensitive information (like passwords, credit cards…

I'll also add the problem in the article seems pretty solvable by allowing user to scope the agentic capabilities to specific websites ( eg "walmart.com:allow_cc,allow_adress").

Re: Scamlexity: When agentic AI browsers get scammed

#19

Post by padolsey »

roxolotl wrote: post #10
benterix wrote: post #5

This should hit the headlines. I was always of the opinion that AI of all kinds is not a threat unless someone decides to connect it to an actuator so that it has direct and uncontrolled effect on the external world. And now it's happening en masse with agents, MCPs etc. I don't even mention things we don't know about (military and other classified projects).

Yea I’ve been surprised about the risk conversations because they cannot do anything if you run them in a sandbox. But it seems like for many the assumed part was we’d hook LLMs into everything asap. It’s absolutely mind boggling.

This is why I've found the safety research conducted by the likes of Anthropic and OAI to be so confusing. Like when they said that models are likely to blackmail developers in order to avoid being 'turned off' [1]. What an utterly and obviously contrived and inevitable derivation of narratives from humans (science fiction and others) in the corpus. Nothing surprising or interesting. However, their hypothesis is presumably(??) that a bad completion from an LLM leads to a bad action in the real world, even though what counts is, as the OP says, the actuators or levers to harm.

Actual LLM completions are moot. I can convince an LLM its playing chess. It doesn't matter as long as the premise is innocuous. I can hook it up to all manner of real world levers. I feel like I'm either missing something HUGE and their research is groundbreaking or they're being performative in their safety explorations. Their research seems like what a toddler would do if tasked with red-teaming AI to make it say naughty words.

EDIT/Addendum: The only safety exploration into agentic harm that I value is one that treats the problem exactly the same as we've been treating cybersecurity vectors. Defence in depth. Sandboxing. Principle of least privelege, etc.

[1] https://www.anthropic.com/research/agentic-misalignment

Re: Scamlexity: When agentic AI browsers get scammed

#20

Post by jdiff »

zenoprax wrote: post #13
hliyan wrote: post #7

Earlier quoted context omitted.

Ah, it's like the good old days when operating systems like DOS didn't really make the distinction between executable files and data files. It would happily let you run any old .exe from anywhere on Earth. Viruses used to spread like wildfire until Norton Antivirus came along.

How is `curl virus.sh | bash` or `irm virus.ps | iex` any different?

You can't easily convince a remote computer to curl | bash itself. Worms spread because remote code execution was laughably easy back then. Also because computer hygiene was abysmal.

LLMs are more than happy to run curl | bash on your behalf, though. If agents gain any actual traction it's going to be a security nightmare. As mentioned in other comments, nobody wants to babysit them and so everyone just takes all the guardrails off.

Post reply on HN