> the implied-but-never-actualized threat of physical violence is
the thing that makes in-person customer service succeed? [emphasis mine]
It's not the thing, but I think it's one of the things, among others such as the ones you listed.
> there is basically no threat
Recent events such as shopkeepers or security guards being assaulted for requiring people to wear masks suggests otherwise. The threat is still very small, but I'd argue it exists and humans aren't always reasonable when it comes to evaluating risk probabilities and will probably overestimate the actual risk and adjust their behavior.
> society built laws and norms that prevent that from being a viable solution
People don't always act reasonably and the perpetrator being punished down the line doesn't change the fact you got your face smashed and needed to get stitches.
> But I'd argue that those extremely rare cases are further proof that the in-person "threat of physical violence" does not play a part in making customer service work well.
The other advantages of in-person communication (which you raised) is that both sides can "read the room" and adjust their behavior before the situation escalates to violence. A customer service person might be more likely to bend the rules when they are in front of a person that's visibly angry or violent so that the situation does not actually escalate that far.