> However, you seem to be stating (correct me if I'm wrong) that there are really two categories of situations here - situation A, where someone is an Asshole with a capital A, and situation B where you have a neurodiverse or immigrant candidate that is misinterpreting the situation and should not be dismissed based on their behavior.
Well, to be honest I think a lot of "capital A assholes" actually are neurodiverse – if not in the more common sense of ASD/ADHD/etc, then in the sense of narcissistic personality disorder, antisocial personality disorder, etc. But, there are limits to the extent of neurodiversity anyone can handle: schizophrenia is a form of neurodiversity too, but a candidate who exhibits florid symptoms of psychosis in a job interview is obviously not a suitable candidate at this time (but they might make an excellent candidate in the future if their psychosis remits). If someone's behaviour is somewhat outside the norm, but they show some insight, willingness to take on feedback and attempt to change, etc, they can still make a positive contribution to a team. If they appear to have no insight, no ability or willingness to consider the feedback of others, no ability or willingness to change or compromise or meet others half-way – it probably isn't going to work out.
> However, based on my (admittedly anecdotal, but extensive) experience situation A is probably something like 95% of the time and situation B is probably something like 5% of the time.
Do you actually know the split is really 95% vs 5%? Is it possible that, some of that 95% really belongs to the 5% but you didn't realise it?
A couple of strategies which might help: (1) force yourself to stop and think about other possible explanations, and whether you have any reason to think any of those other explanations could apply in that particular case; (2) try to gently broach the subject with the candidate, and try to see if you can get any information on the interaction from their perspective. Possibly, their behaviour may seem less unreasonable when you understand it from their point of view. Also, you can judge how open they are to receive feedback and responding to it – there is a big difference between "yeah, I could have handed that situation better" and a rigid insistence that they did nothing wrong.
I'm sceptical of intentionally putting any kind of "hidden test" in a hiring process. When the process is totally transparent – directly ask relevant questions about technical skills/knowledge/experience, soft skills, etc – I think that is fairest on the candidate, and minimises the risk of unintentional indirect discrimination–and I think lawyers who specialise in employment discrimination law would probably agree with that. I don't think that means one has to completely ignore any incidental feedback outside of the transparent formal process, but there is a big difference between incidental feedback and intentionally engineering a "hidden test". Also, I don't think one should rule out an otherwise good candidate on the basis of such incidental feedback without some serious thought and investigation.