It's a good question. Ideally, in a perfect world, your entire computer would be nothing but monstrously fast CPU registers.
But because building those (and the associated support architecture) are much more expensive than slower solutions, we've ended up with a stack of slower and cheaper memories to handle datasets that are too large for the register space, then too large for RAM, etc.
Right now data has to be moved up that stack of faster, but smaller, memories before it can be worked on (nonvolatile storage like a HD->volatile storage like RAM->CPU register) and then back down that stack to store the result (register->RAM->HD).
That movement has to happen across the system bus which on CPU bound operations is a word length for that system -- familiar numbers like 8-bits, 16-bits, 32-bits, 64-bits, etc. but for some older systems that work length might have been something oddball like 7-bits, or 36-bits or something. In other words, you don't move 2GB of data in one shot from your hard drive into RAM. You have to do it in chunks of "words" which means chunks of 32-bits or 64-bits at a time -- over and over again until you've moved the data into memory.
However, RAM is hideously slow compared to CPU registers. I don't know the numbers off the top of my head, but lets say moving data from one part of RAM to another takes 160 cpu cycles (80 to read, 80 to write). This sounds impressively fast on a modern 3.6GHz computer, but that means you can only move 22.5 million "words" around.
By comparison, moving data from one register to another might take 1 cpu cycle.
Furthermore, you can't operate on data in RAM, you have to move it into a register anyways, do the operation, then move the result back out into RAM. So we might be restricted to slightly less than 22.5 million operations per second -- which is pathetically slow.
Clever compilers (and ASM coders) will try and keep things in register space as long as possible to avoid this and try and reach closer to 1 operation per CPU cycle. And modern CPUs have a number of enhancements that also help with this (pipelining, instruction optimization) etc.
But the most important bit are caches. Cache memory are designed to take fewer operations to store/retrieve data than RAM and transparently sandwich in between register space and RAM space and hold working sets of data too large to fit into register space, but are still being worked on so shouldn't end up in RAM space yet. For sake of argument, let's say it takes 20 cpu cycles to read/write from a Cache. If we're operating on data that can fit in a cache, then we can do 180 million instructions per second instead of 22.5. But again, if the working data is too big to fit into the cache, we get a cache miss and end up having to go back to RAM. But now we have to add 20 cpu cycles to a data round trip giving us 180 cycles to read that data and write it back out.
Because the speed difference between RAM and CPUs are so great, multi-level caches which are slightly larger, but use slightly more cycles to retrieve data than the main cache (L1), sit below the CPU cache stack. Even if it takes 30 cycles to read/write some data from an L2 cache (which would only happen on a cache miss in the L1 cache), 20+30 is still faster than 180 for RAM. And so son. Today we have L1, L2 and L3 caches which are all designed to keep the CPU from waiting on RAM.
In other words, all designed to try and overcome the delays of moving data across the system bus and into register space as introduced in the von Neuman architecture.
The system stack these days is something more like HD->RAM->L3->L2->L1->registers with the caveat that moving from one level of the stack to the other probably requires it to move across the system bus which again is likely 32-bits or 64-bits these days. That's 4 or 8 bytes at a time.
With today's multi-gigabyte datasets, that's a ton of data moving across the bus, little of which will end up fitting into any number of caches, slowly eeking it's way into the register space so the CPU can do something like adding two number together. At 3+Ghz, the CPU is mostly just sitting around waiting on data to make it's way up or down this ridiculous stack that's all been designed to accommodate the von Neuman design.
If we had only one kind of memory, and it was fast, and the CPU could directly operate on any part of that memory like it was register space, and we could eliminate the difference between slow but large non-volatile memory (HD) and faster but smaller volatile memory (RAM) computers wouldn't do 180million instructions per second on a good day, they'd do closer to 3.6billion like they are capable of.