People bring this up a lot. It's fair to ask. I do often find it comes from a place of wishful thinking. I mean I'd like to have FTL at my disposal true. But what we're dealing with now
seem to be either hard physical limits or such close approximations to be effectively the same. The key two are:
1. The speed of light is a hard limit; and
2. Thermodynamics (particularly the Second Law) applies. Specifically, you can't get energy from nothing.
From this you can draw a few conclusions:
1. The likely sources of future energy are solar (most likely IMHO) and nuclear fusion. I hope fusion is viable. I'm not yet convinced it is. More exotic far-future options include antimatter and black holes. Antimatter is something you'd make. Think of it like a battery. So you still need the energy to make it. Black holes as propulsion are the same way. Generating power around a large black hole is theoretically possible but has a bunch of issues;
2. Reaction mass is a huge problem for traveling large distances, so much so that using photos to impart momentum seems the most practical. Fusion could be viable here. Antimatter and black hole propulsion are theoretically possible;
3. Because of energy demands, capturing the Sun's energy makes the most sense and doesn't require any "magical" or exotic science or technology;
4. Because of thermodynamics, eventually heat will radiate into space. It's really the only way of dissipating heat, ultimately.
5. Like I said, the IR signature is purely a function of temperature; and
6. A star with a Dyson Swarm will have a very strange (to us) spectrum. Very little visible light. High amounts of IR. There is really no hiding such a megastructure.
So this is essentially a natural conclusion based on physical laws we have no evidence for that they're false or meaningfully incomplete. Thing is, if the speed of light isn't a hard limit that actually makes it more likely we'd find spacefaring life not less because the reach of such a civilization would be so much farther.