Earlier quoted context omitted.
I first came across an alternative view in Godel, Escher, Bach, where Hofstadter was writing about Euclid's parallel postulate and the development of non-euclidean geometry. The latter could have been accepted sooner, he suggests, if people were not hung up on the idea that geometry was about physical space, rather than a derivation from axioms. Ironically, it turned out later that physical space is not euclidean.
I believe our ideas are formed in our brain, that is part of the physical word. So I think the schism you presented would be void.
I think I can avoid the whole issue by observing that I can imagine things that do not physically exist, and they are not made to physically exist by dint of a conceptual representation of them physically existing in my brain.