If you assume that we live in an FTL universe subject to the laws of thermodynamics (as I do), mass and energy (being basically the same thing as we know) are the ultimate limiting factor. This basically means bigger is better. Remember such a civilization is likely thinking on time lines where a trillion years is a blink of an eye.
So how will a spacefaring civilization evolve? Likely into a Dyson swarm for three primary reasons:
1. It requires no new physics. It's essentially an engineering problem that only requires materials as complex as stainless steel and energy tech no more complex than solar power. If you get better tech (eg fusion, graphene for construction) this outcome becomes more likely not less;
2. Orbitals are a highly efficient way of creating living area. A full Dyson swarm around our Sun would likely consume 3. The energy output of the Sun is essentially "free".
A full Dyson swarm makes you a Kardashev-2 (K2) civilization. On the Carl Sagan scale, that means you're consuming ~10^26 Watts of power. And that's just for one Sun. Now consider:
1. Such a civilization is capable of sterilizing the galaxy in 100,000 years if they choose to;
2. There's really no hiding from such a civilization;
3. There's really no hiding a Dyson swarm from a Dyson swarm from an advanced civilization due to the IR signature it would produce. This has nothing to do with say radio communications.
Remember with the Fermi Paradox you don't have to establish what the average civilization does. Say there are 5 spacefaring civilizations in the Milky Way. What are the odds that all of them hide? Now imagine there are 100. The odds of them all hiding are much lower. And it only really takes one who decides to build Dyson Swarms and expand beyond their star to be detectable.
So I can imagine a splinter of a civilization decides to "hide". A good example of this would be if fusion becomes viable. One could sink habitats into Neptune, which is not particularly dense, and probably live there undisturbed and unfound for eons is quite high. That's probably more hidden than, say, trying to hide between stars.
It's just not likely that as the number of civilizations increases that all of them go that route.
My view is that spacefaring life is just quite rare. We may well be the only one within out light cone within the Milky Way. Why that is is an interesting topic.