Instead of self censoring and deleting our past, we should solve the problem on the recipient's side, and harden ourselves against this kind of attack. When someone is booted out of office for the wrong reasons (even though it is "the right person", i.e. someone who's views I don't agree with) I speak out against it. If I was a politician, I would pledge before elections to never step down due to anything that happen…
It's not that people think you need to pay for your past "transgressions" (in their eyes). The problem is that people think they can extrapolate your future decisions, behavior, etc. based on a snippet from your past.
So it's not really about something that happened in the past. It's about something that people will think will happen in the future.