Thanks, this answers my main concerns. I guess that same level of validation applies also for the wallet software we download: we need to make sure the app comes from a source that people have agreed upon, and that usually means applying some sort of general intuition on where to find stuff (googling, forums, reddit etc.). And as you mention, the same trust-game is applied to finding the right chain.
But then, I wonder, if we're all anyway looking for the trustworthy data ourselves (based on general human intuition), why use blockchain at all? Why not just apply the same level of trust towards, say, a regular database?
My thinking is that PoS makes it easier to trust because we'd only have to trust the beginning of the chain, and the rest is verified via the PoS math. Whereas with a normal database we'd have no real way of verifying it hasn't been tampered with?
It does seem a bit scary though that so much money is going into a system where "you have to find the trustworthy data yourself based on your own social network".