51–60 of 154 posts
The best comparable would be the first big computer disaster or the first big internet disaster. Do we have consensus on what that would be?
Longer-term there's an argument to be made for political destabilization driven by social media. We are still seeing this play out.
If everyone no longer trusts any video or image they see (or has plausible deniability to dismiss it), that can lead to a terrible place without a "big disaster" ever taking place.
[flagged]
> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…
A knife won't use 10x human persuasiveness to encourage you to kill yourself.
> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…
Cooking has proven its utility a long time ago. AI girlfriends? Let's say, it's still on the fence.
Also safety: Modern cooking usually occurs in a context of safety-measures developed over thousands of years of injuries, deaths, and burnt cities. Some are so pervasive and traditional that we barely notice, like "no fires unless you have a way to extinguish it in a hurry", or "if you wait too long to eat it is becomes dangerous again."
> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…
But why if a software shop had to do 10% of the regulations and inspections that every restaurant or slaughterhouse has to do, the industry would revolt? Be a libertarian if you want and say neither should be regulated, but that's a 90/10 issue.
Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/ec…
I'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild. I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.
Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/ec…
Sooner or later there'll be a searchable database where you search by name and get their porn history or some other embarassing information.
But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself.
It wasn't until a investigative journalist started going through the citations, and reaching out to the researcher / scientist, to get his opinion, that it was discovered. The researchers cited in the report immediately told the journalist that they had never written or published those cited papers.
Makes one work what other stuff has happened around the globe, where lazy policy makers or bureaucrat have just phoned in their work by using ChatGPT to write a report...or worse, in bad faith used those tools to make up fake research to support their goals.
If everyone no longer trusts any video or image they see (or has plausible deniability to dismiss it), that can lead to a terrible place without a "big disaster" ever taking place.
-clergy and nobles circa 1500