I have a somewhat similar issue. He was never really present in my life (working too much etc.) and so we never connected. But he's also just a... somewhat unpleasant person to be around. If it's more than a few hours, it starts to get too much, so I rarely stay overnight in my home town. Recently (last Christmas) he's been making an effort to connect with me, but it's just so awkward and makes visits for me even more unpleasant. I just can't imagine ever connecting with him, no matter how hard he tries. (Doesn't help that he's recently started smoking again after like 15 years of being clean. I just don't understand it.)
It's most pronounced in my relationship with my father, but it's also my family in general. I feel a bit more connected to my mom and my grandparents, but I just can't say my family is very important to me, even if it sounds sad. My friends have always been my substitute family, in a way, even as a child, and it's even stronger now as a single adult. I probably spend more times with my friends from Norway (I live in Germany) in a single year than with my family in a decade.
Needless to say, this is also an unresolved issue for me. Not sure if it's possible to solve it other than accepting how it is.