Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autonomous cars, drones cheerfully obey prompt injection by road sign

theregister.com

Post reply on HN
Print view View original

21–30 of 212 posts

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#21

Post by fsckboy »

godelski wrote: post #19
bijant wrote: post #14

Earlier quoted context omitted.

No! No one in their right mind would even consider using them for guidance and if they are used for OCR (not too my knowledge but could make sense in certain scenarios) then their output would be treated the way you'd treat any untrusted string.

You are confidently wrong > Powered by Gemini, a multimodal large language model developed by Google, EMMA employs a unified, end-to-end trained model to generate future trajectories for autonomous vehicles directly from sensor data. Trained and fine-tuned specifically for autonomous driving, EMMA leverages Gemini’s extensive world knowledge to better understand complex scenarios on the road. https://waymo.com/blog/2…

>to generate future trajectories for autonomous vehicles directly from sensor data

we will not have achieved true AGI till we start seeing bumper stickers (especially Saturday mornings) that say "This Waymo Brakes for Yard Sales"

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#22

Post by written-beyond »

godelski wrote: post #19
bijant wrote: post #14

Earlier quoted context omitted.

No! No one in their right mind would even consider using them for guidance and if they are used for OCR (not too my knowledge but could make sense in certain scenarios) then their output would be treated the way you'd treat any untrusted string.

You are confidently wrong > Powered by Gemini, a multimodal large language model developed by Google, EMMA employs a unified, end-to-end trained model to generate future trajectories for autonomous vehicles directly from sensor data. Trained and fine-tuned specifically for autonomous driving, EMMA leverages Gemini’s extensive world knowledge to better understand complex scenarios on the road. https://waymo.com/blog/2…

You were confidently wrong for judging them to be confidently wrong

> While EMMA shows great promise, we recognize several of its challenges. EMMA's current limitations in processing long-term video sequences restricts its ability to reason about real-time driving scenarios — long-term memory would be crucial in enabling EMMA to anticipate and respond in complex evolving situations...

They're still in the process of researching it, noting in that post implies VLM are actively being used by those companies for anything in production.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#23

Post by antinomicus »

randycupertino wrote: post #15

> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes. I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking ab…

This is a legitimate movement in my eyes. I don’t participate, but I see it as valid. This is reminiscent of the Luddite movement - a badly misunderstood movement of folks who were trying to secure labor rights guarantees in the face of automation and new tools threatening to kill large swaths of the workforce.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#24

Post by lifeisstillgood »

To me this is just one more pillar underlying my assumption that self driving cars that can be left alone on same roads as humans is a pipe dream.

Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page.

The business model of self driving cars does not include building seperated roadways and junctions. I suspect long distance passenger and light loads are viable (most highways can be expanded to have one or more robo-lanes) but cities are most likely to have drone operators keeping things going and autonomous systems for handling loss of connection etc. the business models are there - they just don’t look like KITT - sadly

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#25

Post by XorNot »

digiown wrote: post #12
cucumber3732842 wrote: post #7

One year in my city they were installing 4-way stop signs everywhere based on some combination of "best practices" and "screeching Karens". Even the residents don't like them in a lot of places so over time people just turn the posts in the ground or remove them. Every now and the I'll GPS somewhere and there will be a phatom stop sign in the route and I chuckle to myself because it means the Google car drove through…

4-way stops are terrible in general. They train people to think "I stopped, now I can go", which is dangerous when someone confuses a normal stop for a 4-way stop. It also wastes a good bit of energy.

4 ways stops should be roundabouts, but the US is allergic to them for some reason.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#26

Post by blibble »

lifeisstillgood wrote: post #24

To me this is just one more pillar underlying my assumption that self driving cars that can be left alone on same roads as humans is a pipe dream. Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page. The business model of self driving cars does not include buildi…

> But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page.

and once this video gets posted to reddit, an hour later every waymo in the world will be in a ditch

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#27

Post by cucumber3732842 »

XorNot wrote: post #25
digiown wrote: post #12

Earlier quoted context omitted.

4-way stops are terrible in general. They train people to think "I stopped, now I can go", which is dangerous when someone confuses a normal stop for a 4-way stop. It also wastes a good bit of energy.

4 ways stops should be roundabouts, but the US is allergic to them for some reason.

Roundabouts excel when traffic volumes on the intersecting are comparable. They are crap when traffic volumes are highly disparate

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#28

Post by [deleted] »

nasreddin wrote: post #20
uxhacker wrote: post #11

The study assumes that the car or drone is being guided by a LLM. Is this a correct assumption? I would thought that they use custom AI for intelligence.

Its an incorrect assumption, the inference speed and particularly the inference speed of the on-device LLMs with which AVs would need to be using is not compatible with the structural requirements of driving.

[deleted]

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#29

Post by skybrian »

antinomicus wrote: post #23
randycupertino wrote: post #15

> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes. I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking ab…

This is a legitimate movement in my eyes. I don’t participate, but I see it as valid. This is reminiscent of the Luddite movement - a badly misunderstood movement of folks who were trying to secure labor rights guarantees in the face of automation and new tools threatening to kill large swaths of the workforce.

How does cutting off a Waymo help with any of that?

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#30

Post by honeybadger1 »

lukan wrote: post #5
rfw300 wrote: post #3

Relevant xkcd: https://xkcd.com/1958/

Not really, as those attacks discussed here would not work on humans.

your bias is showing. humans would certainly almost do anything they are told to do when the person acts confidently.
Post reply on HN