Earlier quoted context omitted.
We're talking about cache, where the addresses of blocks in cache do not have a 1:1 mapping with addresses in memory. Read the section on associativity: http://en.wikipedia.org/wiki/CPU_cache
You get to choose your associativity, you could build a direct-mapped 1GB L1 cache if you wanted. More fundamentally, a memory's access time grows with its capacity because the average and worst-case distances from a bitcell to the read/write port are longer.
Put the GPU and VRAM on the stack and all you need is a couple HDMI ports going out of the chip.