Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autonomous cars, drones cheerfully obey prompt injection by road sign

theregister.com

Post reply on HN
Print view View original

91–100 of 212 posts

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#91

Post by kps »

bigbadfeline wrote: post #41
randycupertino wrote: post #15

> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes. I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking ab…

These drones have cameras, it's a matter of time before they "share" footage... basically becoming robo-cops, traffic edition - this might be of interest to your coworker.

I want to work on artificial road rage. It'd be fun. Don't forget, if you cut off one Waymo, you've cut off every Waymo.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#92

Post by rkomorn »

kps wrote: post #91
bigbadfeline wrote: post #41

Earlier quoted context omitted.

These drones have cameras, it's a matter of time before they "share" footage... basically becoming robo-cops, traffic edition - this might be of interest to your coworker.

I want to work on artificial road rage. It'd be fun. Don't forget, if you cut off one Waymo, you've cut off every Waymo.

There's probably a game to be made out of this called Fleet Wars.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#93

Post by whoiskevin »

theamk wrote: post #74
nharada wrote: post #36

Earlier quoted context omitted.

I think the assumption is valid. Most of the reasoning components of the next gen (and some current gen) robotics will use VLMs to some extent. Deciding if a temporary construction sign is valid seems to fall under this use case.

But unless you are using a single, end-to-end model for the entire driving stack, that "proceed" command will never influence accelerator pedal. Sure, there will be a VLM for reading the signs, but the worst it'd be able to output is things like "there is a "detour" sign at (123, 456) pointing to road #987" - and some other, likley non-LLM, mechanism will ensure that following that road is actually safe.

Not a "proceed" command but they can influence the accelerator. I had a dodge ram van that would constantly decelerate on cruise control due to reading road signs. The signs in some states like California for trucks towing trailers are 55 mph but the speed limit would be 65 or 70 mph. The cruise control would detect the sign and suddenly decelerate to 55.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#94

Post by 47282847 »

Mountain_Skies wrote: post #46
XorNot wrote: post #25

Earlier quoted context omitted.

4 ways stops should be roundabouts, but the US is allergic to them for some reason.

Even rural Georgia has double roundabouts now. Not sure why people on the internet can't contain their glee at stating the US is "allergic" to them when the frequency of roundabouts has grown significantly in recent decades.

Allergies only show when there is something to cause the irritation. Without irritant no allergy.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#95

Post by geekifier »

XorNot wrote: post #25
digiown wrote: post #12

Earlier quoted context omitted.

4-way stops are terrible in general. They train people to think "I stopped, now I can go", which is dangerous when someone confuses a normal stop for a 4-way stop. It also wastes a good bit of energy.

4 ways stops should be roundabouts, but the US is allergic to them for some reason.

There are 10,000+ roundabouts in the US and the number is growing rapidly. One could argue they may even be overused in certain areas (exhibit: Carmel, Indiana).

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#96

Post by nostrademons »

godelski wrote: post #19
bijant wrote: post #14

Earlier quoted context omitted.

No! No one in their right mind would even consider using them for guidance and if they are used for OCR (not too my knowledge but could make sense in certain scenarios) then their output would be treated the way you'd treat any untrusted string.

You are confidently wrong > Powered by Gemini, a multimodal large language model developed by Google, EMMA employs a unified, end-to-end trained model to generate future trajectories for autonomous vehicles directly from sensor data. Trained and fine-tuned specifically for autonomous driving, EMMA leverages Gemini’s extensive world knowledge to better understand complex scenarios on the road. https://waymo.com/blog/2…

This strikes me as a skunworks project to investigate a technology that could be used for autonomous vehicles someday, as well as score some points with Sundar and the Alphabet board who've decreed the company is all-in on Gemini.

Production Waymos use a mix of machine-learning and computer vision (particularly on the perception side) and conventional algorithmic planning. They're not E2E machine-learning at all, they use it as a tool when appropriate. I know because I have a number of friends that have gone to work for Waymo, and some that did compiler/build infrastructure for the cars, and I've browsed through their internal Alphabet job postings as well.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#97

Post by orbital-decay »

Wait, what did just happen here?

1. Some guys did a trivial prompt injection attack, said "imagine if a driverless vehicle used this model", and published it. No problem, someone has to state the obvious.

2. The Register runs this under the clickbait title pretending real autonomous cars are vulnerable to this, with the content pretending this study isn't trivial and is relevant to real life in any way.

I knew The Register is a low quality ragebait tabloid (I flag most of their articles I bother to read), but this is garbage even for them.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#98

Post by InfraScaler »

geekifier wrote: post #95
XorNot wrote: post #25

Earlier quoted context omitted.

4 ways stops should be roundabouts, but the US is allergic to them for some reason.

There are 10,000+ roundabouts in the US and the number is growing rapidly. One could argue they may even be overused in certain areas (exhibit: Carmel, Indiana).

There are more than 15k only in Spain. 10k in the US is nothing.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#99

Post by InfraScaler »

paulclinger wrote: post #39
XorNot wrote: post #25

Earlier quoted context omitted.

4 ways stops should be roundabouts, but the US is allergic to them for some reason.

Roundabouts are great (we just had two complex intersections with traffic lights replaced by roundabouts and the traffic flow is much better), but they take significantly more space than a 4-way stop.

Not necessarily. They could be just painted and barely take any room.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#100

Post by jessa0 »

It sounds like this is a poisoning attack, which has been shown to be pretty trivially defeated [1]. That said, while poisoning countermeasures in the facial recognition case were shown to easily generalize, we dont know yet how general of a defense could be built for a VLM. Which means holding a 0day poisoning attack on a VLM could cause a lot of trouble / deaths before an update to the model with counter-training could be deployed..

[1] https://arxiv.org/abs/2106.14851

Post reply on HN