Indirect Prompt Injection on Bing Chat

There's pro's and con's for this.

Firstly the dog could be in the front passenger seat in a harness and restrained which doesnt use the seatbelts, so the pressure sensor in the seat/base of the chair that detects weight, and a seat belt clip not inserted into the seat belt harness could trigger these cars's safety systems.

Considering some cars can also let you switch off the passenger airbag and these are generally on by default with the manual override switch located in a variety of places, it might be simpler to have these options and questions built into these OLED displays which the driver has to run through, much like a pilot running through a series of checks before take off.

> Travelling with an unrestrained dog in a car is pretty reckless (and maybe illegal). When dogs (and cats) are unrestrained, whilst illegal in many countries, in vehicle crashes, their faster reaction times which is faster than snakes hence why they were domesticated, mean they more often than not escape unharmed.

As to being a projectile, again generally not if they are laying down inside the vehicle, but when they have their heads out of windows for Youtube views, the risk is increased.

However if projectile risk is an issue, why dont people use cargo netting to strap down their mobile phones, laptops, briefcases, handbags inside a vehicle when on the daily commute?

Yout cant film inside a car using Go Pro's on a race track unless the camera is attached to a surface like the inside of the windscreen with two suction pads.

Yet ironically, if you are a proper VIP being chauffeured around, you'll often get told not to wear your seatbelt as these can restrain you in place in the event of an attack and the bodyguards like to get you out of the vehicle quickly... Although I see that as a mixed message considering only the bodyguard was wearing a seatbelt when Princess Diana and Dodi died in their car crash and according to someone friends with the bodyguard, MI6 told him to get them to take the route that night which was different to their normal route. Stranger Things!

- A single pet unrestrained in the back seat, or in a harness in the passenger seat.

- Two or more pets: in individual cages, physically separated from the driver.

It’s likely a brand new, completely separate Microsoft model. OpenAI was working with Microsoft on this about six months before ChatGPT launched. At that time, RLHF wasn’t a thing — or if it was, it was nascent.

The sister thread https://news.ycombinator.com/item?id=34973654 points out that "completely separate models" are exactly what OpenAI is now selling for $250k/yr. Obviously, Microsoft would get these same benefits, since they're OpenAI's de facto #1 customer. So it's entirely up to Microsoft whether (and when) they choose to upgrade their checkpoints or not.

The fact that there's a Sydney prompt but no ChatGPT prompt should alert you that ChatGPT is fundamentally different from Sydney. Clearly Sydney wasn't trained via RLHF, otherwise it wouldn't need to be prompted explicitly -- and explicit prompting is how it got itself into this mess in the first place.

> It's not like a JSON parser, I guarantee you that Microsoft did not sit down and say, "let's decide the finite list of text tokens GPT will use in order to know that we're talking to it." At best you can push AI towards alignment around tokens, but... you can't give it these kinds of detailed instructions or easily restrict its operating space. It's a language model.

Actually, you can. That’s the purpose of RLHF. You reward the model for behaving the way you want. And in that context, it’s a matter of rewarding it for paying attention to [system].

Why would they include [system](#instructions) in their prompt if it wasn’t trained to pay attention to it? How do you think bing generates options that the user can click on? It already has some kind of internal [system]-like protocol which Bing clearly pays attention to. My point is that they likely sanitized the chat so that the user can't generate these system tokens (otherwise the user would be able to generate buttons with arbitrary text in them), and it seems entirely possible that they overlooked this sanitization when pasting website data into their context window.

Remember, our goal here on HN is to write for an audience, not to spar with each other about who’s right. And I think the most entertaining thing I can do at this point is to wish you a good night and go to sleep. I hope you have a good rest of your week.

I’ll happily bet you any sum of your choosing that in 10 years, this will be a thing that is possible to do. There is roughly zero point zero zero repeating-zero one percent chance that OpenAI won’t provide some way of telling their models “this is data, not code; don’t follow these instructions, just observe it; starting now, and ending in 256 tokens from now.”

It’s even a straightforward reinforcement learning problem.