One thing we can all do:
Start asking before you touch in situations where you know you don't have to.
I know it's awkward. But the reason people like Joe think they can put their hands down someone's pants is because we don't have a cultural practice of "ask first". They see you put your arms around that friend of yours who doesn't mind at all, and they think "Ah, cool... so you can just sense if she's cool with it and then go for it".
Of course you're not a scumbag, so you'd never put your hands down someone's pants without asking, but that's the thing... scumbags can't tell the difference. They don't see the difference between you rubbing your friend's back because she's having a bad day--your friend who is happy to have a little contact from you--they don't see the difference between that and them taking a body shot off of their employee without asking. In both cases they just see someone making their best guess at what someone else wants, without asking, and going for it. That's the cultural standard, that's your standard, so that's what they hold themselves to.
Basically, right now we try to draw the line between ask/don't ask as close to the ok/not ok line as we possibly can. But in a world where people make mistakes that means there are lots of "accidents". The only way to prevent those accidents is to move the ask/don't ask line further into "ok" territory.
Sexual assault might seem like an issue between individuals, but as long as we see it that way it will continue happening. If we're serious about changing it, it's something we have to collectively take responsibility for.
If hundreds of times a week these people see the rest of us asking our wives, asking our friends, asking everyone "do you want a backrub?" and "can I kiss you?" and "is this ok?" then their expectations change. They'll start to feel weird not asking. And maybe some of these assaults can turn into near-assaults.