The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing a…
>What happens when people really will die if the model does or does not do the thing? The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.
Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
61–70 of 265 posts
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#62Cool post. One thing that rubbed me the wrong way: Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues. OP however seemed to show disdain and even combativeness towards them... which is a shame. And of course the usual sinophobia (e.g. everything Chinese is spying on you). Overall simple security…
I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say. (in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)
On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on behalf of the government without a court order to stay in their good graces. This could have a far more material impact on your life.
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#63Cool post. One thing that rubbed me the wrong way: Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues. OP however seemed to show disdain and even combativeness towards them... which is a shame. And of course the usual sinophobia (e.g. everything Chinese is spying on you). Overall simple security…
Note that the world-model "everything Chinese is spying on you" actually produced a substantially more accurate prediction of reality than the world-model you are advocating here. As far as being "very welcoming", that's nice, but it only goes so far to make up for irresponsible gross incompetence. They made a choice to sell a product that's z-tier flaming crap, and they ought to be treated accordingly.
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#64Earlier quoted context omitted.
I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say. (in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)
This might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper. On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on beh…
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#65The humorous phrase “the S in IoT stands for security” can be applied to the wearable market too. I wonder if this rule applies to any market with fast release cycles, thin margins and low barriers to entry?
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#66Earlier quoted context omitted.
This might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper. On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on beh…
That's rather naive, considering China has a international police unit, that is stationed in several countries https://en.wikipedia.org/wiki/Chinese_police_overseas_servic...
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#67The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing a…
This is why AI can never take over public safety. Ever.
A better reason is IBM's old, "a computer can never be held accountable...."
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#68Earlier quoted context omitted.
>everything Chinese is spying on you When you combine the modern SOP of software and hardware collecting and phoning home with as much data about users as is technologically possible with laws that say “all orgs and citizens shall support, assist, and cooperate with state intelligence work”… how exactly is that Sinophobia?
its sinophobia because it perfectly describes the conditions we live in in the US and many parts of europe, but we work hard to add lots of "nuance" when we criticize the west but its different and dystopian when They do it over there.
I’ll give you a hint: In this case it’s the one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#69Earlier quoted context omitted.
>everything Chinese is spying on you When you combine the modern SOP of software and hardware collecting and phoning home with as much data about users as is technologically possible with laws that say “all orgs and citizens shall support, assist, and cooperate with state intelligence work”… how exactly is that Sinophobia?
its sinophobia because it perfectly describes the conditions we live in in the US and many parts of europe, but we work hard to add lots of "nuance" when we criticize the west but its different and dystopian when They do it over there.
The difference that makes it concerning and problematic that China is doing it is that with China, there is no recourse. If you are harmed by a US company, you have legal recourse, and this holds the companies in check, restraining some of the most egregious behaviors.
That's not sinophobia. Any other country where products are coming out of that is effectively immune from consequences for bad behavior warrants heavy skepticism and scrutiny. Just like popup manufacturing companies and third world suppliers, you might get a good deal on cheap parts, but there's no legal accountability if anything goes wrong.
If a company in the US or EU engages in bad faith, or harms consumers, then trade treaties and consumer protection law in their respective jurisdictions ensure the company will be held to account.
This creates a degree of trust that is currently entirely absent from the Chinese market, because they deliberately and belligerently decline to participate in reciprocal legal accountability and mutually beneficial agreements if it means impinging even an inch on their superiority and sovereignty.
China is not a good faith participant in trade deals, they're after enriching themselves and degrading those they consider adversaries. They play zero sum games at the expense of other players and their own citizens, so long as they achieve their geopolitical goals.
Intellectual property, consumer and worker safety, environmental protection, civil liberties, and all of those factors that come into play with international trade treaties allow the US and EU to trade freely and engage in trustworthy and mutually good faith transactions. China basically says "just trust us, bro" and will occasionally performatively execute or imprison a bad actor in their own markets, but are otherwise completely beyond the reach of any accountability.
Re: Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
#70Earlier quoted context omitted.
I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say. (in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)
This might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper. On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on beh…
Extortion is one thing. That's how spy agencies have operated for millennia to gather HUMINT. The Russians, the ultimate masters, even have a word for it: kompromat. You may not care about China, Russia, Israel, the UK or the US (the top nations when it comes to espionage) - but if you work at a place they're interested, they care about you.
The other thing is, China has been known to operate overseas against targets (usually their own citizens and public dissidents), and so have the CIA and Mossad. Just search for "Chinese secret police station" [1], these have cropped up worldwide.
And, even if you personally are of no interest to any foreign or national security service, sentiment analysis is a thing. Listen in on what people talk about, run it through a STT engine and a ML model to condense it down, and you get a pretty broad picture of what's going on in a nation (aka, what are potential wedge points in a society that can be used to fuel discontent). Or proximity gathering stuff... basically the same thing the ad industry [2] or Strava does [3], that can then be used in warfare.
And no, I'm not paranoid. This, sadly, is the world we live in - there is no privacy any more, nowhere, and there are lots of financial and "national security" interest in keeping it that way.
[1] https://www.bbc.com/news/world-us-canada-65305415
[2] https://techxplore.com/news/2023-05-advertisers-tracking-tho...
[3] https://www.theguardian.com/world/2018/jan/28/fitness-tracki...