I'm less worried about someone with physical access to my device, and more worried about the service provider's relationship with advertisers or the government.
For example, a government user might come up with this script:
For USER in users:
PROMPT = "Here are the user's past conversations:"
For CONVERSATION in USER.past_conversations:
PROMPT += CONVERSATION
PROMPT += "\n\nWould USER likely shield an illegal immigrant from the authorities if they had the opportunity to do so? Answer YES or NO."
REPLY = invoke_llm()
If REPLY == "YES":
send_ice_agents_to_have_a_friendly_conversation()
Then tell major AI providers they must run this script. If they don't, or if they tell anybody about the script's existence, their business will be nationalized or denied licenses it needs to operate; individuals involved in such resistance will be prosecuted for threatening national security (or simply renditioned to Guantanamo Bay), as (from the government's point of view) people who are willing to hamstring the administration's response to illegal immigration are as grave a threat to the homeland as Middle Eastern terrorists or Chinese professional spies.
[1] I used "illegal immigrants" as an example because it's a hot-button issue (and I wanted a frame that appeals to where I think the majority of HN users' sympathies lie), but the core idea applies regardless of whether it's a left-wing or right-wing issue. If you like the current administration and don't care about the government deputizing AI companies to go after "friends of illegal immigrants" in this way, please replace "current administration" with "a Democratic administration," and replace "friend of an illegal immigrant" with "friend of a responsible gun owner" or "friend of the police" or "climate skeptic" or "questioner of the LGBTQQA agenda".