The solution might be easy: recognize liability.
When a human cajoles someone into killing themselves, they probably get a criminal trial, as well as a civil lawsuit from the family.
Imagine a company set up an advice telephone line, and staffed it entirely with humans who were known clinical bullshitters. The company knows each call has a significant chance of the staffer simply making something up, or saying something absolutely crazy. The company markets the clinical bullshitters as geniuses, and somewhere there's the most mild disclaimer they could craft, about how the staffer might say something incorrect.[*]
Every time one of their staff persuades someone to kill themselves... that company is looking at criminal and civil liability.
The company absolutely knew this would happen, and that it will keep happening, but they are betting that they can keep getting away with this and other "externalities", and become millionaires or billionaires. Before legislators and courts realize that, just because the company inserted this new-fangled telephone technology into the interaction, that doesn't mean that existing laws don't apply to the interaction.
[*] Analogous to the tiny gray smallprint search engine LLM summary one that isn't even shown unless you click to see more, and then only after you scroll all the way to the bottom of that and look for it. Or the chat UI tiny gray smallprint that says you should "double-check" (no explanation what that means), and makes even this weak disclaimer not even text-selectable, so even if you see it, it's lost when you copy&paste the LLM response. Understated, and in what looks like bad faith. On the telephone in our example, it's a single tiny beep, at and the end of the call, that's actually the words "This may not be correct" sped up 3x, and whispered, and is often missed because the customer hangs up, but that's by design.*