Was that really "you"? Maybe just like "you" but with horns. Or just the top half, and lasted just long enough to shoot. Or, nothing like you, really. And then your grandmother spontaneously conceived an exact copy of your father.
Or, almost infinitely more likely, who you thought was your grandfather wasn't, and somebody shot him but nobody told you. And, somebody else who looked like him showed up and your grandmother married him and didn't tell that either.
So you need a more reliable sort of own-grandfather-killing than mere possibility affords.