The idea of panspermia isn't a new one. There are several variations.
The very early universe was relatively warm. Simple or even complex life could've evolved and spread through parts of the Universe with relative ease given the much shorter distances. It seems highly unlikely though given the rarity of metals (in the astronomical sense; meaning anything other than hydrogen and helium) and the likely needed timescales. This probably means at best it was organic molecules of some level of complexity.
There are later versions of this where panspermia occurred much more recently, either intentionally or not.
Whatever the case it's all unprovable speculation. If we ultimately the same unlikely patterns repeating on different wrolds then I guess we can revisit.
So how recent in this hypothesis did this occur? There are fossil records for people going back millions of years. Are we talking all life? Or just people? If it's all life then we need to go back billions of years, at which point we're just talking about spreading amino acids. Such a connection will be similar to our fraternal bond with bananas.
> However at some point between the tech to travel between stars and full Dyson spheres
This seems unlikely. Why? Because the tech for a Dyson Swarm is basically stainless steel, solar panels and the ability to build things in space in large quantities. The last one is significant of course but is largely gated on the high cost of getting things into orbit, which is something likely to plummet in the coming centuries.
Saying we could build a self-sustaining orbital within 1,000 years I don't think is a stretch. Only industrial scale separates building 1 and building a million.
But there's another problem: people often neglect the energy cost of interstellar travel. It's... massive. And this is even assuming you solve the reaction mass problem. Interstellar travel almost seems predicated on a Dyson Swarm simpsly to have sufficient energy.
> ... also concerning as a fast growing civilization that contravenes their anti-growth norms.
This is a well-trodden avenue of thought on the Fermi Paradox. What if alien civilizations just stop growing? If there's 1, sure that might be possible. But what if there were 1,000? Would they all follow this path? It gets increasingly unlikely that not one would grow significantly. Non-growth civilizations would be at an extreme disadvantage with another who has vastly more energy and matter at their disposal. Like you almost have to grow just in case someone else does.
But all of this just seems like highly selective curve overfitting to reach the desired conclusion that UFOs are alien in origin.