> If planets become indefensible
Planets are indefensible, as is most of space if your orbit is predictable. All you need is a body bigger than your margin of computational error, and reaction mass.
> exponential growth
My point is that in order for civilization to be multi-starred, they'd need to communicate. Once a new star is colonized, due to the constraint on light speed, they become two separate civilizations. This means each star basically starts from the beginning, each time. So, if you estimate a couple of million years for a civilization to just to consider expanding... it'd take awhile. A lot more than a few million years to colonize the galaxy because not every star will consider expanding.
> the reason is resource competition.
Resources for what, exactly? Populations self-govern growth. It isn't always pretty, since over-population means a lot of deaths. But hungry people aren't going to say "I'm hungry so I'm going to build a space ship and hunt Space Cows. Who's coming with me?"
Further, there is no evidence that dinosaurs left this planet. They had 165 million years to do so. So, it doesn't 'just take' a few million years and there is no guarantee that a civilization will expand beyond a star. Sure, you could argue that dinosaurs probably weren't intelligent, but there's no evidence for that other than a small cavity size for brains; which doesn't necessarily mean they were stupid. Whales have bigger brains than humans, but I don't see them leaving the earth any time soon either.
My point is, even if you decide to colonize another planet, you are effectively cut off from your home. Technology WILL be lost since you basically have to restart manufacturing from nothing, with a small population at that, and there are a whole different set of filters to pass through. The odds of that civilization growing to the point of wanting to move on, without dying out itself, is remarkably low.
Interestingly, colonization of stars beyond a certain distance might even be impossible without some kind of artificial gravity or cryo. Especially if it requires multiple generations to arrive.