I mostly agree with your take, but I'll pick on this sentence:
> Given the huge benefits of a Dyson Swarm and the fact that it seems to require no exotic materials and no more physics it looks increasingly likely that we are very alone in the Milky Way
The fact that we can come up with the idea of a Dyson Swarm doesn't mean that not finding any in our observations results in we "being alone in the galaxy". It's also Main Character Syndrome, in a way.
Even discarding the idea of alien life being so different from ours that we wouldn't recognize it even if we were looking right at it, and assuming a "human like civilization", it's perfectly possible for there being unknown physics to us that make the idea of a Dyson Swarm unnecessary. Using our own civilization as an example, in the 1950s and 60s we did all of our data broadcasting over radio waves, and built huge powerful antennas that screamed about our presence to the wider universe. People then thought "well, if we're broadcasting all this stuff, where are the alien broadcasts? why can't we hear them? we must be alone in the galaxy". Fast-forward to now. Our current tech allows pretty much all communications to be over cables, and we're being much less wasteful with our emmiting; our radio emissions are diminishing over time.
So, not a century has passed, and already newer technology has proven our assumptions of alien life wrong. Why would it be any different with Dyson Swarms? You can't know how more advanced technology looks like, you can only extrapolate with what we have now.