HPC admin here, generally serving "long tail of science" researchers.
In today's x86_64 hardware, there's no "supercomputer memory subsystem". It's just a glorified NUMA system, and the biggest problem is putting the memory close to your core, i.e. keeping data local in your NUMA node to reduce latencies.
Your resource mapping is handled by your scheduler. It knows your hardware, hence it creates a cgroup which satisfies your needs and as optimized as possible, and stuffs your application into that cgroup and runs it.
Currently king of high performance interconnects is Infiniband, and it accelerates MPI at the fabric level. You can send messages, broadcasts and reduce results like there's no tomorrow. Because when the message arrives you, it's already reduced. When you broadcast, you only send a single message which is broadcasted at fabric layer. Multiple Context IB cards have many queues and more than one MPI job can run on the same node/card with queue/context isolation.
If you're using a framework for GPU work, the architecture & optimization is done at that level automatically (the framework developers do the hard work generally). NVIDIA's drivers are pure black magic, too. They handle some parts of the optimization, too. InterGPU connection is handled by a physical fabric, managed by drivers and its own daemon.
If you're CPU bound, your libraries are generally hand tuned by its vendor (Intel MKL, BLAS, Eigen, etc.). I personally used Eigen, and it has processor specific hints and optimizations baked in.
The things you have to worry is to compile your code for the correct architecture, make sure that the hardware you run on can satisfy your demands (i.e.: do not make too many random memory accesses, keep the prefetcher and branch predictor happy if you're trying to go "all-out fast" on the node, do not abuse disk access, etc.).
On the number crunching side, keeping things independent (so they can be instruction level parallelized/vectorized), making sure you're not doing unnecessary calculations, and not abusing MPI (reducing inter-node talk to only necessary chatter) is the key.
It's way easier said than done, but when you get the hang of it, it becomes like a second nature to think about these things, if these kinds of things are your cup of tea.