And that's when the Olympics actually occur, and the host city is able to gain some revenue from visitors coming and spending a considerable amount of money.
If Tokyo builds the required infrastructure and then no one comes, I've got to think future host cities will re-think this madness.
Personally I'd like to see a distributed Olympics, with mostly pre-existing facilities being used. Maybe swimming happens in Copenhagen, basketball in LA, beach volleyball in Phuket, etc. Maybe some events could be held in smaller countries or cities that had never been able to complete for a full Olympics before. A larger percentage of the world population would be local to some part of the Olympics, perhaps for once highlighting sameness rather than differences.
Or screw it, we could just let dumb governments keep throwing piles of money at this nonsense.