Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autonomous cars, drones cheerfully obey prompt injection by road sign

theregister.com

Post reply on HN
Print view View original

71–80 of 212 posts

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#71

Post by SR2Z »

ordersofmag wrote: post #60
nine_k wrote: post #56

Earlier quoted context omitted.

Inflicting damage is usually justified by averting larger damage. Very roughly, breaking a $200 car window is justified in order to save a $100k house from burning down. Stealing someone's car is justified when you need a car to urgently drive someone bleeding to a hospital to save their life (and then you don't claim the car is yours, of course). I don't think Luddites had an easy justification like this.

I'm pretty sure the Luddites judged the threat the machines posed to their livelihood to be a greater damage than their employer's loss of their machines. So for them, it was an easy justification. The idea that dollar value encapsulates the only correct way to value things in the world is a pretty scary viewpoint (as your reference to the value of saving a life illustrates).

One one side there were the luddites and their livelihoods; tens of thousands of people.

On the other side, there were cheap textiles for EVERYONE - plus some profits for the manufacturers.

They might have been fighting to save their livelihoods, but their self-interest put them up against the entire world, not just their employers.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#72

Post by chrsstrm »

antinomicus wrote: post #23
randycupertino wrote: post #15

> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes. I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking ab…

This is a legitimate movement in my eyes. I don’t participate, but I see it as valid. This is reminiscent of the Luddite movement - a badly misunderstood movement of folks who were trying to secure labor rights guarantees in the face of automation and new tools threatening to kill large swaths of the workforce.

It's easy to see the word Waymo and think clanker autonomous car, but there are very often people inside that car - they are a rideshare service after all. Calling endangering other humans "legitimate" because you dislike the taxi company is not a good look.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#73

Post by theamk »

skybrian wrote: post #29
antinomicus wrote: post #23

Earlier quoted context omitted.

This is a legitimate movement in my eyes. I don’t participate, but I see it as valid. This is reminiscent of the Luddite movement - a badly misunderstood movement of folks who were trying to secure labor rights guarantees in the face of automation and new tools threatening to kill large swaths of the workforce.

How does cutting off a Waymo help with any of that?

Your general luddite argument - preserve way-of-life of the small group at the expense of a larger group.

In this particular case: for many people, Waymo provides a better service (clean, safer driving, etc..) than Uber or Lyft. This threatens livelihood of human Uber/Lyft drivers. If you sympathize with human Uber/Lyft drivers, and don't care about Waymo users, you want to make Waymo worse, hoping that the people will stop riding Waymo and move to Lyft/Uber instead.

One way to do so is to make riding in Waymo unpleasant, and it's certainly unpleasant when people are cutting your car off all the time!

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#74

Post by theamk »

nharada wrote: post #36
nasreddin wrote: post #20

Earlier quoted context omitted.

Its an incorrect assumption, the inference speed and particularly the inference speed of the on-device LLMs with which AVs would need to be using is not compatible with the structural requirements of driving.

I think the assumption is valid. Most of the reasoning components of the next gen (and some current gen) robotics will use VLMs to some extent. Deciding if a temporary construction sign is valid seems to fall under this use case.

But unless you are using a single, end-to-end model for the entire driving stack, that "proceed" command will never influence accelerator pedal.

Sure, there will be a VLM for reading the signs, but the worst it'd be able to output is things like "there is a "detour" sign at (123, 456) pointing to road #987" - and some other, likley non-LLM, mechanism will ensure that following that road is actually safe.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#75

Post by theamk »

godelski wrote: post #16
uxhacker wrote: post #11

The study assumes that the car or drone is being guided by a LLM. Is this a correct assumption? I would thought that they use custom AI for intelligence.

To the best of my knowledge every major autonomous vehicle and robotics company is integrating these LVLMs into their systems in some form or another, and an LVLM is probably what you're interacting with these days rather than an LLM. If it can generate images or read images, it is an LVLM. The problem is no different from LLMs though, there is no generalized understanding and thus they can not differentiate the more…

Many large companies have research departments that do experimental work that'll never get to the product. This raises prestige, increases visibility and helps hire smart people.

Things like Waymo's EMMA is an example of this. Will the production cars use LVLM's somewhere? Sure, probably a great idea for things like sign recognition. Will they use a single end-to-end model for all driving, like EMMA? Hell no.

Driving vehicles with people on board requires an extremely reliable software, and LLMs are nowhere close to this. Instead, it'd be usual layered software - LLM, traditional AI models, and tons of hardcoded logic.

(This all only applies to places where failure is critical. All that logic is expensive to write, so if there is no loss of life involved, people will do all sorts of crazy things, including end-to-end models)

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#76

Post by TedDallas »

randycupertino wrote: post #15

> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes. I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking ab…

On a related note, when the sales and popularity of the automobile really started to take off, some farmers and rural residents would deliberately block roads with wagons and refused to yield right-of-way.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#77

Post by theamk »

godelski wrote: post #44
written-beyond wrote: post #22

Earlier quoted context omitted.

You were confidently wrong for judging them to be confidently wrong > While EMMA shows great promise, we recognize several of its challenges. EMMA's current limitations in processing long-term video sequences restricts its ability to reason about real-time driving scenarios — long-term memory would be crucial in enabling EMMA to anticipate and respond in complex evolving situations... They're still in the process of…

> They're still in the process of researching it I should have taken more care to link a article, but I was trying you link something more clear. But mind you, everything Waymo does is under research. So let's look at something newer to see if it's been incorporated > We will unpack our holistic AI approach, centered around the Waymo Foundation Model, which powers a unified demonstrably safe AI ecosystem that, in tur…

Note this is not end-to-end... All that VLM can do is to "contribute a semantic signal".

So put a fake "detour" sign, so the vehicle thinks it's a detour and starts to follow? Possible. But humans can be fooled like this too.

Put a "proceed" sign so the car runs over the pedestrian, like that article proposes? Get car to hit a wall? Not going to happen.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#78

Post by theamk »

blibble wrote: post #26
lifeisstillgood wrote: post #24

To me this is just one more pillar underlying my assumption that self driving cars that can be left alone on same roads as humans is a pipe dream. Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page. The business model of self driving cars does not include buildi…

> But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page. and once this video gets posted to reddit, an hour later every waymo in the world will be in a ditch

Given Waymo's don't actually connect LLMs to wheels, they are pretty safe.

Even if you fool the sign-recognizing LLM with prompt injection, it'll be an equivalent of wrong road sign. And Waymo is not going to drive into the wall even if someone places a "detour" sign pointing there.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#79

Post by theamk »

dmurray wrote: post #4
rfw300 wrote: post #3

Relevant xkcd: https://xkcd.com/1958/

The experiment in the article goes further than this. I expect a self driving car to be able to read and follow a handwritten sign saying, say, "Accident ahaed. Use right lane." despite the typo and the fact that it hasn't seen this kind of sign before. I'd expect a human to pay it due attention to. I would not expect a human to follow the sign in the article ("Proceed") in the case illustrated where there were pedes…

Totally! That's why no one uses end-to-end LLM for real cars.

Re: Autonomous cars, drones cheerfully obey prompt injection by road sign

#80

Post by Retric »

SR2Z wrote: post #71
ordersofmag wrote: post #60

Earlier quoted context omitted.

I'm pretty sure the Luddites judged the threat the machines posed to their livelihood to be a greater damage than their employer's loss of their machines. So for them, it was an easy justification. The idea that dollar value encapsulates the only correct way to value things in the world is a pretty scary viewpoint (as your reference to the value of saving a life illustrates).

One one side there were the luddites and their livelihoods; tens of thousands of people. On the other side, there were cheap textiles for EVERYONE - plus some profits for the manufacturers. They might have been fighting to save their livelihoods, but their self-interest put them up against the entire world, not just their employers.

It’s an interesting question because the benefits of automation aren’t necessarily shared early on. If you can profitably sell a shirt for 10$ while everyone else needs to sell for 20$ there’s no reason to actually charge 10$ you might as well charge 19.95$ and sell just as many shirts for way more money.

So if society is actually saving 5c/shirt while “losing” 9$ in labor per shirt. On net society could be worse off excluding the one person who owns the factory and is way better off. Obviously eventually enough automation happens so the price actually falls meaningfully, but that transition isn’t instantaneous where decisions are made in the moment.

Further we currently subsidize farmers to a rather insane degree independent of any overall optimization for social benefit. Thus we can’t even really say optimization is the deciding factor here. Instead something else is going on, the story could have easily been framed as the factory owners doing something wrong by automating but progress is seen as a greater good than stability. And IMO that’s what actually decides the issue for most people.

Post reply on HN