Earlier quoted context omitted.
For everyone in the EU: Copying and pasting sensitive data (like customer data) into AI tools is a violation of the GDPR, and potentially the AI act, which will be enforced soon. These violations come with hefty fines.
You’re brushing too broad a stroke GDPR only affects personal information. There’s plenty of sensitive business information that is not covered by GDPR - for example per business customer revenue data - that is legal to put into an AI tool but your employer may not want you to.
I'd be surprised if there weren't already phishing attacks that work by pretending to be a LLM.