You are talking some decent ideas but I do not agree on the Heracleum mantegazzianum point (we use to call it borschevik).
1. Any invasive specie known today is about transporting an organism, evolutionally willing to live from one place to another. GMO organism aren't evolutionally willing to live, they just genetically willing to give us better tomatoes.
2. Absence of non-GMO is not a problem at all if I still have an access to bleeding-edge reproductionable seeds as it happens today. The problem is in patent system, what if the cost of bleeding-edge seeds will be costing so much that small farmers with expensive and not yummy non-GMO will be out of market? Have you seen any vegetable before selection? Nobody needs the corn with few grains while we use to have a corn with few hundrends of grains. Even the most green-headed crackpot on the planet will not grow that in 21th century.
3. I do not believe that all plants harvested from GMO are _really_ sterile. Since they use to have a reproductive system so all kinds of mutations are possible if something with CRISPR/Cas9 goes wrong. And there are some experiments with attempt to spread some modified genes to something like all the malary mosquitoes [1] . There are some GMO even among humans, and that humans are not sterile [2]
[1] https://news.ycombinator.com/item?id=31095529
[2] https://en.wikipedia.org/wiki/He_Jiankui_affair