Earlier quoted context omitted.
The big bang didn't happen at a single point, it happened everywhere. You can look out from anywhere and see the cosmic background radiation having expanded from your location, wherever that location might be.
Doesn't that imply that, "everywhere" was, precisely, a single point?
One theory that might help you visualize an alternative is that the big bang was basically two 3D universes (floating in higher-dimensional space) slapping against each other really hard. That creates an explosion everywhere even if everywhere is quite big or even infinite.