The simple reality with remote controllers is that there is a lot of extra network latency that pretty much makes any real time controlling of the car impractical. Most of this stuff happens over mobile networks too so there might be package loss, low resolution video. Maybe the video freezes for a second or two occasionally. Etc.
Even if human controllers actually could pay attention 100% of the time, they'd struggle to respond in time to a lot of dangerous situations. Most accidents happen when one of the cars (or their drivers) fails to react in time with what is effectively a split second decision.
Autonomous driving (with or without a controller) means a computer takes essentially all of those decisions for the simple reason that any human controller would probably be too late way too often.
Once you accept that simple logical reality, the role of that controller becomes more clear: they are there to step in and provide instructions to the car when it encounters some challenging situation and slows down, pulls over, or stops in a safe place. This probably doesn't involve any joysticks or steering wheels.
Controller responses are not real time critical. They can't be. It would fail to work too often. Also, most controllers probably need to monitor more than one car. Which only makes the problem worse. And they might have to juggle two stuck cars at once.
Mostly autonomous cars are pretty good at object detection and not crashing into stuff (all the real time stuff). It's object classification and interpreting complex situations where cars get stuck or might sometimes still do dangerous/illegal/sub optimal things. Getting stuck or slowing down is fine. The human controller can fix that. Doing the wrong thing is more problematic.