System vary. I'm most familiar with Ethereum's, which is the biggest one in production.
It starts with a fairly traditional BFT protocol that reaches consensus as long as everyone follows the rules. These have been known in computer science for decades.
The twist is that if anyone breaks the rules, you can prove they did it. So everybody deposits some ETH as a security bond, and if they break the rules, any other staker can submit proof of that to get a small reward, and inflict a penalty on the offender.
An offense by a small amount of stake does no damage and is assumed to be an accident, so the penalty is small. An offense by enough stake to cause problems is completely destroyed so the attack can't continue.