Earlier quoted context omitted.
> People tell these models everything . You don't think a pretty unethical company can figure out a way to monetize that? I assume research-level questions about scientific topics (with a focus on math and computer science) are not easy to monetize (yes, this is by far the most common kind of question to AI models for me). :-D
I guarantee you're a lot easier to manipulate than you think.
I can imagine some hypothetical scenarios, but I do believe that I have strong evidence that this is mostly not the case.
For example, I think I have already told the story how when I worked together with a person who is a business consultant (and thus salesman) told me that C-level executives are an incredibly easy sales target compared to me. For him, selling something to me was "ultra-hard mode", basically because I could immediately see through every sales tactic that he tried on me.