Earlier quoted context omitted.
I'm not assuming anything about bit representations. In this case, the spec language is quite clear and unambiguous. 6.3.2.3 paragraph 7: A pointer to an object type may be converted to a pointer to a different object type. If the resulting pointer is not correctly aligned[footnote 68]) for the referenced type, the behavior is undefined. Otherwise, when converted back again, the result shall compare equal to the orig…
I do not see there a promise that the cast will produce an invalid pointer, nor anything prohibiting the compiler from rounding the pointer down, thus producing a valid one. “Converted” does not require bit copy. I don’t see how this interpretation is against any section of the spec.
In practice performing a cast doesn't really do much until you dereference, but without a carve out in the spec, it does really mean "the behavior is undefined".