I agree with what you're saying here - absolutely, bad medical advice from AI (or anywhere) can cause harm.
What I disagree with is the way that you're drawing the conclusion that there isn't even a valid argument to be made for using AI. To be clear, I didn't say we absolutely should do this - the part of my post you quoted is where I said there's an argument for it (i.e. we should have the discussion and evaluate).
If an AI says take hydroxychloroquine for COVID then absolutely that will cause harm. But that theoretical possibility isn't enough to determine whether an AI should be a source of medical advice - you have to actually evaluate whether/how often it will do that. If .0001% of the time it says to do that and the rest of the time it gives good advice on how to treat COVID at home and what symptoms to look out for that would cause you to need to call an ambulance, then the harms are probably greatly outweighed by the benefits.
This is especially true in the case that I argued - where the alternative is no medical advice, because people don't have access to doctors. Lots of people died at home of COVID when they could have been saved at a hospital - we have to consider what the benefits to them might be in addition to what the detriments of AI giving bad advice could be.
The last thing I'll say is that your position seems to be that if a source of medical advice sometimes gives bad advice, we should not use it. The problem is that based on that position, people should not go to doctors. Doctors are humans who make mistakes - plenty of evidence of that to go around. All of these points that you've made apply to human doctors:
> Giving out wrong information can easily kill people. Or permanently insure them. There's a tremendous amount of harm that can occur if you do things wrong in medicine.
> When you attach "[a doctor] said X" to something, it adds the appearance of authority.
> If you had [a doctor] say "go take hydroxychloroquine for COVID", you'd end up with a lot of people getting harmed
I want to point out that there absolutely were doctors who told patients to take hydroxychloroquine.
This is all to say that I think you have to weigh relative harms and benefits. Should someone with access to top tier medical care get advice from an AI? Almost certainly not. Somebody who has access to no medical care or to the sort of doctor who prescribes hydroxychloroquine? As I said initially, it's not necessarily a hard yes but there is certainly an argument to be made that it's a yes.