Earlier quoted context omitted.
> the C representation closely matches what is actually happening It really doesn't, though. Although your CPU might present system RAM as one contiguous array of bytes to your program, the C compiler follows different rules – see strict aliasing and other pointer dereference rules. For example, the following is Undefined Behavior and your C compiler may or may not generate the assembly you expect: int x = *(int *)0x…
Casting an integer to a pointer is implementation defined, not UB. And every sane implementation does what everyone expects because its how memory mapped IO works (but you probably want a volatile in there and maybe a compiler or memory barrier as well depending on what the hardware guarantees about the access patterns for that particular range of addresses)
You're right, that was a bad example. Here's a better one:
int x, y;
ptrdiff_t diff = &x - &y;
This is Undefined Behavior, because &x and &y don't point to the same object.