Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

blog.mgdproductions.com

Post reply on HN
Print view View original

11–20 of 265 posts

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#11

Post by williamscales »

komali2 wrote: post #9

> "and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you." Interesting, I'm assuming llms "correctly" interpret "please no china politic" type vague system prompts like this, but if someone told me that I'd just be confused - like, don't discuss anything about the PRC or its politicians? Don't discuss the his…

> Don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin?

In my mind all of these could be relevant to Chinese politics. My interpretation would be "anything one can't say openly in China". I too am curious how such a vague instruction would be interpreted as broadly as would be needed to block all politically sensitive subjects.

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#13

Post by mmaunder »

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.”

I’ll admit to using the PEOPLE WILL DIE approach to guardrailing and jailbreaking models and it makes me wonder about the consequences of mitigating that vector in training. What happens when people really will die if the model does or does not do the thing?

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#14

Post by reactordev »

mmaunder wrote: post #13

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing a…

This is why AI can never take over public safety. Ever.

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#15

Post by aspbee555 »

komali2 wrote: post #9

> "and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you." Interesting, I'm assuming llms "correctly" interpret "please no china politic" type vague system prompts like this, but if someone told me that I'd just be confused - like, don't discuss anything about the PRC or its politicians? Don't discuss the his…

it is to ensure no discussion of Tiananmen square

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#16

Post by zihotki »

crtasm wrote: post #7
JohnMakin wrote: post #4

“decrypt” function just decoding base64 is almost too difficult to believe but the amount of times ive run into people that should know better think base64 is a secure string tells me otherwise

>However, there is a second stage which is handled by a native library which is obfuscated to hell

That native obfuscated crap still has to do an HTTP request, that's essentially a base64

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#17

Post by sneak »

reactordev wrote: post #14
mmaunder wrote: post #13

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing a…

This is why AI can never take over public safety. Ever.

https://www.wired.com/story/wrongful-arrests-ai-derailed-3-m...

Story from three years ago. You’re too late.

Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)

#20

Post by ben_w »

mmaunder wrote: post #13

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing a…

> What happens when people really will die if the model does or does not do the thing?

Then someone didn't do their job right.

Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output.

But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die.

In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people.

At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise.

I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.

Post reply on HN